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ABSTRACT 


The  author  evaluates  LeatherNet,  a  Distributed  Interactive  Simulation 
compliant,  virtual  simulation  system  being  developed  by  the  Advanced  Research  Projects 
Agency  to  demonstrate  Modeling  and  Simulation(M&S)  technologies  and  to  partially  fulfill 
the  U.  S.  Marine  Corps  M&S  goals.  The  research  focuses  on  evaluation  of  LeatherNet  as  a 
mission  planning  and  briefing  tool  for  Marine  infantry  company  commanders,  staff,  and 
subordinate  leaders.  Evaluation  is  based  on  user  perception  and  user  performance  on  a  live 
fire  range  subsequent  to  using  the  system.  The  user  surveys  indicate  high  user  acceptance 
and  belief  that  LeatherNet  is  a  valuable  mission  planning  and  briefing  tool  and  that 
LeatherNet  has  a  good  potential  to  be  an  effective  training  tool  for  commanders  and  their 
staffs.  User  performance,  evaluated  by  subject  matter  experts  on  a  live  fiore  range,  showed 
no  statistically  significant  improvement  for  groups  exposed  to  LeatherNet  when  compared 
to  groups  that  did  not  use  LeatherNet.  The  author  eplains  why  true  differences,  even  if  they 
do  exist,  would  be  difficult  to  detect  due  to  the  lack  of  experimental  control  and 
recommends  action  to  be  taken  by  the  Marine  Corps  to  conduct  further  testing  with  greater 
experimental  control.  The  author  also  suggests  steps  the  Marine  Corps  can  take  to  optimize 
its  investment  in  M&S. 
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L  INTRODUCTION 


A.  RESEARCH  OBJECTIVE 

The  purpose  of  this  thesis  is  to  determine  the  training  effectiveness  of  using 
LeatherNet,  a  virtual  simulation  system  that  models  several  Marine  Corps  live  fire  and 
maneuver  ranges,  to  train  infantry  company  command  element  staff  and  subordinate  leaders. 
The  evaluation  specifically  addresses  LeatherNet  as  a  tool  to  conduct  mission  briefing  and 
mission  planning.  Training  effectiveness  is  measured  through  user  performance  evaluations, 
conducted  by  subject  matter  experts  on  live  fire  and  maneuver  exercises,  after  user  exposure 
to  the  LeatherNet  system.  User  perception  is  also  used  to  determine  training  effectiveness 
as  measured  through  user  surveys. 

B.  SCOPE  OF  THIS  STUDY 

The  scope  of  this  evaluation  is  limited  to  testing  the  training  effectiveness  of  using 
LeatherNet  as  a  mission  planning  and  mission  briefing  tool.  To  make  a  comprehensive 
assessment  of  the  training  effectiveness  of  LeatherNet,  with  a  high  degree  of  accuracy, 
would  require  many  factors  not  available  at  the  time  of  this  study.  First,  the  system  is  not 
mature  enough  to  allow  comprehensive  testing  of  some  system  capabilities  such  as 
interactive  force  on  force  war  gaming  for  mission  rehearsal.  Second,  many  features  of 
LeatherNet  that  might  have  been  tested,  were  not,  due  to  user  resistance  to  features  they  felt 
did  not  behave  realistically  or  were  too  time  consuming  to  use.  Thirdly,  fimding  and 
personnel  to  conduct  a  true  experimental  test  could  not  reasonably  be  made  available,  nor 
is  such  a  test  desirable  for  this  particular  stage  of  development  of  LeatherNet.  Determining 
the  training  effectiveness  of  military  simulation  systems  is  difficult,  complex,  and  costly, 
particularly  for  systems  designed  for  collective  training.  Therefore,  the  expense  of 
conducting  a  true  experiment  on  a  demonstration  project  still  under  development  cannot  be 
justified.  The  user  survey  and  user  performance  evaluation  are  intended  only  as  indicators 
of  user  acceptance  and  as  a  preliminary  evaluation  of  the  system's  value  for  future  reference. 
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C.  MILITARY  SIMULATION  DEVELOPMENTS 

Short  of  combat,  all  training,  mission  rehearsals,  and  exercises  conducted  in 
preparation  for  war  can  be  considered  simulation.  With  the  advance  of  computer 
technologies,  many  of  the  functions  of  field  training  can  now  be  further  simulated  using 
virtual  simulation  rather  than  using  live  participants  in  field  exercises  to  simulate  the  actions 
of  combatants.  The  U.S.  military  is  more  reliant  upon  simulation  than  is  generally  realized. 
Development  and  testing  of  new  weapons  such  as  aircraft,  tanks,  and  ships  are  accomplished 
with  extensive  use  of  simulation  to  test  their  value.  Furthermore,  the  crews  of  many  weapons 
systems  now  rely  heavily  on  simulators  for  training,  and  commanders  can  develop  and  test 
battle  plans  and  tactics  on  simulators  using  combat  models.  Simulation  has  the  potential  to 
be  used  for  a  wide  range  of  applications  including  training,  testing,  evaluation,  mission 
rehearsal,  and  system  acquisition  [Ref  1:  p.  1-2]. 

The  U.S.  military  employs  three  forms  of  simulation  to  train  its  forces  and  portray 
what  happens  when  one  military  force  engages  another;  (1)  constructive  simulation  which 
uses  mathematical  models  of  combat  to  simulate  warfare  for  everything  from  a  single 
weapons  system  to  national  warfare,  (2)  live  simulation  involving  real  military  forces  and 
simulated  weapons  effects,  and  (3)  virtual  simulation  using  computers  to  simulate  warfare 
in  synthetic,  computer  generated  environments.  [Ref  1:  p.  1-4] 

1.  National  Defense  Issues 

Since  the  end  of  the  Cold  War,  several  trends  have  provided  impetus  to  increase 
reliance  upon  modeling  and  simulation  within  the  Department  of  Defense  (DoD).  Although 
maintaining  combat  readiness  is  essential  for  the  military,  resources  devoted  to  maintain 
combat  readiness  have  steadily  declined  since  the  demise  of  the  Soviet  threat.  The  decline 
can  be  expected  to  continue  for  the  foreseeable  future  as  the  nation  continues  to  grapple  with 
the  federal  budget  deficit.  Second,  despite  the  diminishing  resources,  the  DoD  has  seen  no 
lessening  of  its  operational  tempo,  and  in  fact,  has  experienced  a  proliferation  of  missions 
and  contingencies.  As  funding  has  diminished,  future  readiness  in  terms  of  recapitalization 
and  modernization  is  at  risk  of  being  sacrificed  to  pay  for  near  term  commitments.  [Ref  2: 

p.  2-6] 
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While  the  threat  of  a  major  EastAVest  confrontation  has  subsided,  smaller  scale 
regional  contingencies  have  multiplied  as  nationalistic  movements,  ethnic  disputes,  and 
rivalries  once  suppressed  by  communist  regimes  continue  to  inflame  the  former  Warsaw 
Pact  countries.  Additional  challenges  are  posed  by  countries  such  as  Iraq  and  North  Korea 
which  threaten  the  security  of  their  neighboring  countries,  while  countries  such  as  Libya, 
China  and  Iran  pose  less  imminent  threats.  In  addition  to  containing  the  spread  of  regional 
disputes,  the  U.S.  military  must  also  be  prepared  to  combat  terrorism  and  the  use  of  nuclear, 
biological,  or  nuclear  agents.  Ongoing  efforts  to  obtain  these  agents  from  the  stockpiles  of 
the  old  Warsaw  pact  countries,  or  to  manufacture  them  in  various  hostile  Third  World 
nations  present  a  significant  and  growing  threat  to  the  U.S.  and  its  allies.  The  combination 
of  these  less  predictable  threats  in  far  flung  areas  of  the  globe  increase  the  difficulty  of 
maintaining  a  military  capability  to  deal  with  them.  [Ref  3] 

As  part  of  the  current  national  military  strategy,  the  military  has  increasingly  been 
used  for  peacetime  engagements  or  Operations  Other  Than  War  (OOTW).  In  order  to 
promote  regional  security  and  stability,  U.S.  forces  have  frequently  been  called  upon  to 
conduct  nation  building,  security  assistance,  humanitarian,  counter  drug/terrorism,  and 
peacekeeping  missions.  Recent  operations  in  Haiti,  Rwanda,  and  Somalia,  as  well  as  the 
current  operations  in  Bosnia  demonstrate  such  OOTW.  [Ref  2;  p.  6-12] 

Not  only  have  resources  diminished,  encroachment  by  civilian  communities  upon 
military  bases,  continued  base  closing  in  the  U.S.,  and  the  loss  of  overseas  bases  and  training 
areas  as  the  U.S.  reduces  its  overseas  military  presence,  have  further  constrained  the 
military's  training  opportunities.  The  loss  of  training  areas  is  further  aggravated  by  increased 
environmental  regulations  and  endangered  species  protection  which  erode  the  military's 
unlimited  access  and  use  of  many  of  its  remaining  training  areas. 

The  increasing  cost  of  training  is  another  constraint  that  limits  the  ability  of  the  DoD 
to  maintain  high  readiness  levels.  As  weapons  systems  have  become  more  sophisticated  with 
the  use  of  precision  munitions  and  high  technology  weapons  platforms,  both  the  cost  of  the 
munitions  and  the  weapons  systems  have  increased  dramatically.  It  is  not  uncommon  for 
Soldiers  and  Marines  to  be  allowed  a  single,  or  even  no,  live  fire  shots  in  a  year's  time  from 
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some  of  the  more  sophisticated  missile  systems  for  which  they  are  crew  members.  Even 
some  of  the  more  common  anti-tank  missile  systems  such  as  the  TOW  and  Dragon  systems 
have  munitions  costs  that  typically  limit  the  expenditure  of  live  missiles  for  training 
purposes  to  one  or  two  missiles  per  year.  Many  weapons  platforms,  such  as  the  Ml  Abrams 
tank  and  the  F-18  aircraft,  are  increasingly  expensive  to  operate,  due  to  fuel  and  maintenance 
costs,  regardless  of  the  amount  of  training  munitions  expended.  The  net  result  of  these  cost 
increases  in  an  era  of  reduced  funding  is  that  the  services  are  often  forced  to  balance 
between  readiness  training  and  investment  in  force  structure.  [Ref  4:  p.  2-4] 

2.  Simulation  Benefits 

As  the  Commandant  of  the  Marine  Corps  pointed  out  in  his  planning  guidance,  in 
an  era  of  diminishing  resources,  history  has  shown  that  Marines  can  survive  both  as 
individuals,  and  as  an  institution,  if  they  remain  highly  trained  and  ready.  He  notes  that  the 
Marine  Corps  has  always  turned  to  training  and  education  systems  to  keep  its  warfighting 
edge.  At  several  points  in  his  guidance  he  makes  reference  to  his  general  statement  that  "the 
use  of  simulation,  virtual  reality  models,  and  various  warfighting  games  can  make 
subsequent  field  training  more  effective."  [Ref  5] 

In  short,  modeling  and  simulation  can  be  for  the  U.S.  Armed  Forces  what  gliders  and 
cardboard  mockup  tanks  were  to  the  German  Wehrmacht  prior  to  World  War  11,  when  more 
attractive  and  realistic  training  options  were  prohibited.  Despite  constraints  being  placed  on 
the  military's  ability  to  conduct  training  in  the  traditional  manner,  large  scale  use  of  virtual 
simulation,  imagination,  and  resourcefulness  in  implementing  the  emerging  modeling  and 
simulation  technologies,  can  help  enable  the  DoD  to  avoid  unpreparedness  by  augmenting 
live  training  and  thus  providing  the  basis  of  future  battlefield  success. 

In  the  context  of  current  and  expected  future  developments  within  the  DoD  and  the 
U.  S.  Marine  Corps,  several  assumptions  have  been  made  by  the  Marine  Corps  Modeling  and 
Simulation  Management  Office  (MCMSMO)  that  drive  the  need  for  increased  reliance  on 
modeling  and  simulation.  The  most  significant  of  these  are  as  follows: 

1 .  That  resources  for  DoD  will  be  constrained  for  the  foreseeable  future. 

2.  That  DoD  will  continue  to  develop  joint  tactics,  exercises  and  operations  to 
include  joint  simulation. 
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3.  That  the  cost  of  weapons  platforms  and  munitions  Avill  continue  to  increase  as  they 
become  more  sophisticated. 

4.  That  training  areas  will  continue  to  have  constraints  imposed  upon  their  use  by 
civilian  communities  and  agencies  [Ref  6:  p.  1-6] 

Despite  diminishing  resources,  the  military's  investment  in  simulation  will  continue 
to  expand.  Within  the  Marine  Corps,  modeling  and  simulation  technologies  will  be  used  for 
many  varied  applications  to  include  validation  of  requirements  and  doctrine,  testing  and 
evaluation,  training  combat  forces,  improvement  of  acquisition  processes,  and  mission 
planning,  preview  and  rehearsal.  [Ref  7] 

The  primary  reason  for  the  use  of  simulation  is  economic.  Virtual  simulation  for 
example,  allows  commanders  and  staffs  to  experiment  without  the  cost  of  fuel,  anununition, 
equipment  maintenance,  and  large  numbers  of  personnel.  Training  can  be  accomplished  at 
a  small  fraction  of  the  cost  of  using  real  systems  in  the  field.  A  single  TOW  anti-tank 
missile  costs  $11,500,  but  a  single  TOW  shot  in  the  Precision  Gunnery  Training  System 
(PGTS)  costs  only  $.05  [Ref  21].  Likewise,  the  cost  of  operating  an  Ml  Abrams  tank  is 
approximately  $92.00  per  mile  and  transit  to  a  gunnery  range  for  a  tank  platoon  can  cost 
thousands  of  dollars,  but  the  cost  of  the  Unit  Conduct  of  Fire  Trainer  (UCOFT)  is  only  $.25 
per  hour.  [Ref  8:  p.  18] 

Many  simulators  such  as  the  UCOFT  provide  training  enhancement  in  addition  to 
the  cost  savings.  Various  studies  of  the  UCOFT  have  found  that  tank  crews  trained  on  the 
system  experience  substantial  gains  in  the  percent  of  targets  acquired,  engaged,  hit,  and 
destroyed  and  that  they  open  fire  with  their  first  round  25%  faster  than  conventionally 
trained  gunners  [Ref  21].  The  Indoor  Simulated  Marksmanship  Trainer  (ISMT)  can 
similarly  improve  training  that  can  be  difficult  if  not  impossible  to  replicate  on  a  real  range. 
For  the  M203  grenade  launcher,  for  example,  the  system  provides  detailed,  accurate 
measurements  of  the  gunners  sight  picture  and  sight  alignment  for  each  shot.  Thus, 
deficiencies  that  would  normally  go  unnoticed  can  be  coached.  [Ref  8;  p.  19] 

Linking  of  various  simulators  allows  training  to  be  distributed  over  great  distance 
at  minimal  cost.  Since  networked  simulators  share  common  databases,  participants  using 
the  simulators  can  "see"  the  same  visual  images  from  their  relative  positions  in  the  terrain 


5 


database.  Entities,  that  represent  individuals,  units,  vehicles,  or  aircraft  can  see  the  same 
targets,  and  each  other  regardless  of  the  location  of  the  simulator,  and  more  importantly,  the 
personnel  participating  in  the  simulation.  The  result  is  sigmficant  cost  saving  by  avoiding 
the  need  to  physically  bring  forces  together  to  train  together. 

As  an  indicator  of  the  savings  possible,  in  the  1992  REFORGER  (Return  of  Forces 
to  Germany)  exercise,  the  Army  used  constructive  simulation  to  avoid  34  million  dollars  in 
costs  as  compared  the  same  exercise  in  1988  without  simulation  [Ref.  9].  The  1992  exercise 
allowed  more  commands  to  participate  in  the  event,  although  the  number  of  troops 
participating  in  the  live  exercise  was  reduced.  The  net  result  was  that  many  units  had  the 
benefit  of  participating  in  an  exercise  without  the  expense  of  transporting  all  the  personnel 
and  equipment  typically  necessary. 

Virtual  simulators  provide  the  benefit  of  allowing  the  user  to  experience  the  virtual 
model  of  the  real  world  he  would  not  otherwise  be  able  to  experience  due  to  fiscal 
constraints,  occupation  by  hostile  forces,  or  simply  being  too  dangerous  for  peacetime 
operations  [Ref  9].  Using  the  virtual  model,  the  user  can  freely  navigate  the  terrain,  viewing 
the  environment  from  both  the  fiiendly  and  enemy  perspectives,  conduct  mission  planning, 
and  conduct  war  gaming  against  a  sjmthetic  enemy,  to  mention  just  a  few  of  the  options 
available  to  the  user. 

With  the  growth  of  environmental  concerns  upon  our  existing  training  areas,  not  to 
mention  the  loss  of  training  areas,  modeling  and  simulation  technologies  offer  a  viable 
alternative  while  ensuring  the  protection  of  both  endangered  species  and  encroaching 
civilian  communities.  Constructive  and  virtual  simulation  require  little,  if  any,  deployment 
of  troops  to  the  field,  and  they  can  act  as  a  mechanism  to  reduce  the  size  of  the  live 
simulation  force  without  sacrificing  the  training  objectives.  Virtual  and  constructive 
simulations  leave  behind  no  dud  ordnance,  fuel  spills,  or  environmental  degradation  like  that 
typically  resulting  from  large  scale  field  exercises. 

3.  Simulation  Limitations 

The  benefits  of  virtual  simulation  are  many  but  several  drawbacks  or  potential 
drawbacks  do  exist.  The  possibility  exists  that  as  the  use  of  simulation  increases,  reduction 
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in  training  on  the  actual  equipment  and  reduced  participation  in  live  simulations  can  also 
reduce  both  the  user's  cortBdence  in  the  use  of  the  actual  equipment  and  the  military 
readiness  of  the  force.  Additionally,  if  the  behaviors  of  a  simulator  do  not  accurately  model 
the  real  world,  the  simulator  may  instill  habits  that  are  incorrect  for  use  with  real  scenarios 
and  equipment.  Accurate  models  of  the  characteristics  of  weapons  systems,  individual 
combatants,  command  and  control  structures,  various  sensors,  and  weather,  to  mention  a 
few  of  the  elements  requiring  modeling  in  military  simulators,  are  difficult  to  attain. 
Furthermore,  the  modeling  data  may  not  exist  or  may  be  prohibitively  expensive  to  develop. 
The  use  of  funds  for  development  and  procurement  of  simulation  systems  of  unknown  value 
can  be  a  dubious  undertaking  that  has  the  potential  to  detract  from  military  readiness.  In 
addition,  the  value  of  a  simulation  system  cannot  easily  be  determined  until  the  system 
reaches  mature  stages  of  its  development  and  user  testing  takes  place,  which  is  after  the  bulk 
of  the  development  costs  have  been  incurred.  Finally,  the  potential  exists  for  the  compromise 
of  security  if  access  is  gained  to  networks  that  support  interactive  simulation,  exposing 
sensitive  tactics  and  weapons  capabilities.  [Ref  1;  p.  1-14] 

4.  Simulation  Technologies 

Several  factors  are  driving  the  need  for  ever  greater  reliance  upon  virtual  simulation. 
As  previously  mentioned,  the  ability  to  conduct  live  simulation  will  be  fiscally  constrained 
in  an  era  of  military  downsizing,  exerting  pressure  to  reduce  the  size  and  number  of  live 
simulations.  Reduction  in  the  size  and  number  of  available  training  areas,  and  the  inability 
to  adequately  simulate  weapons  effects  on  live  personnel  and  equipment  are  also  prime 
factors  driving  the  need  for  virtual  simulation. 

Perhaps  the  most  significant  driver  increasing  the  role  of  simulation  is  the  growth 
of  simulation  capabilities  and  related  technologies.  These  new  technologies  are  rapidly 
changing  the  manner  in  which  the  U.S.  military  trains,  deploys,  and  executes  its  missions. 
Precision  munitions,  satellite-based  surveillance  and  communication  systems,  and 
computer-based  mission  planning  systems  are  but  a  few  examples  that  are  dramatically 
increasing  the  information  processing  capabilities  and  have  served  to  expedite  decision 
making  and  execution  on  the  battlefield.  The  rapid  growth  in  computer  capabilities. 
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including  advances  in  computer-based  graphics  and  communication  technologies,  have 
enabled  the  use  of  networks  of  interactive  simulators  to  distribute  training  over  great 
distances  [Ref  10:  p.  14-15],  Today's  simulators  can  realistically  model  military  entities, 
environments,  and  combat  against  synthetic  opposing  forces.  Once  individuals  and  units 
have  been  trained  on  the  real  weapons  systems  and  tactics,  further  training  can  be  conducted 
on  such  current  systems  as  the  Army's  SIMNET  (Simulator  Networking)  using  the  emerging 
communication  systems  such  as  the  Distributed  Interactive  Simulation  (DIS)  network  to  link 
simulators.  Such  networked  simulators  are  designed  for  unit  and  joint  training  in  large  scale, 
two-sided  engagements  with  various  weapons  t5^es  and  command  and  control  groups. 
Networked,  interactive  simulation  systems  currently  being  developed  and  fielded  include 
both  British  and  German  versions  of  SIMNET,  the  Close  Combat  Tactical  Trainer  (CCTT), 
and  the  Synthetic  Theater  of  War  (STOW)  demonstration  project.  [Ref.  1:  Chap.  I,  p.  10] 

a.  Distributed  Interactive  Simulation 

DIS  is  a  follow-on  development  of  SIMNET  technology.  Once  fully 
developed,  DIS  will  support  real  time  communications  between  networks  of  various  types 
of  simulators,  in  multiple  locations,  to  create  a  realistic  synthetic  environment  to  perform 
interactive  training,  design  and  prototyping,  war  gaming,  and  other  applications.  The 
infi-astructure  will  be  designed  to  allow  interactive  sessions  between  simulation  systems  from 
all  the  services,  built  for  separate  purposes,  with  different  technologies,  and  from  different 
vendors.  DIS  exercises  will  incorporate  a  mixture  of  live,  virtual,  and  constructive  entities 
and  allow  units  of  all  sizes  to  engage  in  two-sided  combat  whether  in  a  simulator  or  using 
actual  equipment  on  an  instrumented  range.  Interoperability  in  DIS  is  established  through 
the  use  of  standard  protocols,  communication  architectures,  interface  standards,  and  other 
standardized  elements  necessary  to  link  the  various  simulators  into  a  seamless  synthetic 
environment.  [Ref  12:  p.  1-7] 

With  DIS,  the  military  will  not  only  retain  the  use  of  stand  alone  simulators, 
but  will  be  able  to  conduct  "Total  Force"  training  without  requiring  units  to  leave  their  home 
bases.  As  such,  a  cost  savings  will  be  gained  from  avoiding  the  transportation  of  personnel 
and  equipment  that  would  normally  be  required  to  facilitate  an  operation  and  the  reduction 
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of  wear  and  tear  on  the  equipment.  Additionally,  DIS  provides  a  safe  environment  where 
operation  and  contingency  plans  can  be  developed,  tested,  and  evaluated  in  a  realistic 
simulated  environment.  Fielding  of  deployable  simulators  aboard  ship  and  at  forward 
locations,  and  the  use  of  satellite  communications,  will  enable  exercising  Marine  Air-Ground 
Task  Forces  (MAGTF)  though  the  use  of  simulation.  Feedback  from  real  simulations  into 
virtual  simulators  will  be  accomplished  through  the  continued  development  of  instrumented 
ranges  that  instrument  personnel,  weapons,  and  targets  to  provide  data  on  events  in  both  time 
and  space,  between  opposing  sides  in  simulated  combat.  [Ref  7;  Chap.  1,  p.  5] 

If  real  time  interaction  between  simulated  crewed  platforms  is  to  be  achieved, 
many  issues  must  be  resolved  that  are  not  addressed  in  the  commercial  market.  To  enable 
DIS  to  operate  in  real  time,  a  dedicated,  high  capacity  communication  network  with  transfer 
rates  higher  than  155  megabytes/second  will  be  required,  and  must  include  land  sea  and  air 
connectivity  [Ref  7;  p.  3-35].  The  network  must  be  capable  of  conveying  data  to  interested 
simulation  nodes  with  low  latency  of  100-300  milliseconds  and  low  latency  variance.  The 
latency  standard  refers  to  the  maximum  allowed  time  for  data  to  travel  from  the  transmitting 
application  to  the  receiving  application.  DIS  requires  that  each  simulated  entity  inform  all 
other  simulated  entities  of  its  activities  without  knowing  who  or  what  the  capabilities  of  the 
other  entities  are.  To  handle  this  many  to  many  communication,  the  network  must  support 
multicast  addressing  rather  than  point  to  point  addressing.  The  Defense  Advanced  Research 
Projects  Agency  (DARPA)  has  sponsored  the  development  of  the  Defense  Simulation 
Internet  (DSI)  as  a  high  capacity,  general  purpose,  packet  switching  Wide  Area  Network 
(WAN)  to  achieve  this  goal.  Another  challenge  is  handling  the  immense  amount  of  data 
arriving  at  each  node  and  separating  only  the  data  pertinent  to  that  node  from  the  data  stream 
for  processing  at  that  node.  [Ref  12:  p.  23-29] 

Technological  developments  critical  to  simulation  and  training  that  are 
associated  with  DIS  include  networks,  semi-autonomous  forces  (SAPOR),  terrain  and 
environment  databases  and  display,  range  instrumentation,  and  individual  combatants  [Ref 
1:  Chap.  6,  p.  2-19].  Infrastructure  developments  required  to  support  these  efforts  include 
developing  hardware  suites  for  data  processing  and  display  for  the  instrumented  ranges  and 
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developing  hardware  suites  for  mission  preview  capability  [Ref.  6;  p.  3-9],  User  interface 
development  for  virtual  simulation  will  focus  on  three  dimensional  displays,  SAFOR  that 
correctly  model  personnel  and  weapon  capabilities,  and  SAFOR  construction  tools  capable 
of  rapidly  building  synthetic  environments  to  promote  rapid  scenario  and  mission  generation. 
[Ref  7;  Sec.2,  p.  9-17] 

D.  THESIS  OUTLINE 

Chapter  II,  Military  Simulation  Systems,  provides  an  overview  of  military 
simulations  efforts  within  both  the  DoD  and  the  U.S.  Marine  Corps.  Major  modehng  and 
simulation  (M&S)  initiatives  from  both  organizations  are  discussed  to  provide  a  background 
on  the  major  simulation  developments,  challenges,  and  objectives.  Finally,  the  LeatherNet 
project  is  described  to  provide  an  overview  of  its  purpose,  the  development  strategy,  and  the 
expected  benefits  of  the  system. 

In  Chapter  III,  the  methods  for  conducting  training  and  cost  effectiveness  analysis 
of  simulation  systems  are  examined.  The  challenges  to  performing  such  analysis  are 
discussed  to  provide  an  explanation  of  how  the  methodology  for  evaluating  LeatherNet 
was  developed. 

Chapter  IV,  Methodology,  explains  the  purpose  and  methodology  for  evaluating 
LeatherNet.  The  methodology  for  data  collection  and  analysis  is  described  for  both  the  user 
survey  and  the  user  performance  evaluation.  The  test  constraints  and  environment,  to 
include  the  threats  to  the  vahdity  of  the  experiment,  are  also  examined. 

The  results  of  the  data  analysis  are  discussed  in  Chapter  V.  The  results  are  displayed 
graphically  in  bar  charts  and  further  explained  in  the  text.  Various  data  analysis  techniques 
are  used  to  analyze  the  user  survey  and  user  performance  evaluation  results. 

In  the  final  chapter.  Conclusions  and  Recommendations,  the  meaning  of  the  data 
analysis  results  and  their  implication  on  the  LeatherNet  system  are  discussed. 
Recommendations  for  future  research  in  evaluating  LeatherNet  and  similar  simulation 
systems  that  are  used  for  collective  training  are  presented.  Additionally,  suggestions  are 
made  regarding  steps  the  Marine  Corps  can  take  to  promote  the  use  of  M&S  technologies 
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and  to  ensure  the  Marine  Corps  remains  a  relevant  participant  in  M&S  development. 
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n.  MILITARY  SIMULATION  SYSTEMS 


A.  ASSESSMENT  OF  MODELING  AND  SIMULATION  IN  DOD 

1.  Background 

The  M&S  developments  prior  to  1990  were  characterized  by  fragmentation  and 
limited  coordination  across  the  Services  and  functional  communities.  In  recogmtion  of  these 
deficiencies,  the  DoD  created  the  Defense  Simulation  Management  Office  (DMSO)  to 
facilitate  the  coordination  of  M&S  activities  within  and  across  the  different  communities. 
Additionally,  many  of  the  DoD  components  established  their  own  M&S  organizations  to 
coordinate  their  efforts  within  their  own  component  and  with  the  other  services.  These  steps 
have  fostered  advances  in  architecture,  standards,  and  protocols;  representation  of  the 
environment,  weapons  systems,  and  human  behavior;  and  the  fielding  of  M&S  systems  and 
associated  infrastructure.  [Ref  13;  Chap.  3] 

2.  Architectures,  Standards,  and  Protocols 

Interoperability  and  reuse  of  simulation  systems  are  limited  within  the  DoD  due  to 
the  lack  of  a  common  framework  for  simulation  architectures.  It  is  widely  recognized  that 
a  common  framework  must  be  established  to  facilitate  the  interoperability  of  all  the  various 
simulators.  To  meet  this  objective,  DIS  protocols  and  standards  have  been  implemented  to 
establish  a  common  data  exchange  environment,  and  they  continue  to  be  refined.  There 
remains  the  need  to  significantly  expand  DIS  and  develop  its  architecture  to  support  a 
broader  range  of  capabilities  such  as  representing  command  and  control  more  realistically, 
supporting  simulations  with  different  time  management  methods,  and  reducting  the 
computational  and  communication  bandwidth  demands. 

The  Aggregate  Level  Simulation  Protocol  (ALSP)  confederation  of  models  currently 
in  use  will  remain  the  primary  tool  for  simulation  for  joint  level  trmning  until  the  Joint 
Simulation  System  (JSIMS)  reaches  its  Initial  Operating  Capability  (IOC).  ALSP  has  been 
critical  for  constructing  "federations"  of  existing  models  for  connecting  theater-level  tactical 
simulators  supporting  a  Avide  spectrum  of  joint  and  combined  exercises,  such  as 
REFORGER.  This  confederation  of  models,  however,  has  only  limited  interoperability,  takes 
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a  long  time  to  set  up,  and  requires  many  personnel  to  operate.  DIS,  as  a  part  of  JSIMS,  will 
replace  ALSP.  [Ref  14:  p.  1147] 

3.  Representation  of  the  Environment,  Systems,  and  Human  Behavior 

The  simulated  terrain  representations  achieved  to  date  are  impressive,  but  they  are 

resource  intensive  to  produce  and  are  largely  non-reusable  by  different  simulators.  The  DoD 
continues  to  conduct  research  and  development  for  reusable  terrain  and  environmental 
databases,  but  is  impeded  by  a  lack  of  clear  understanding  of  the  modeling  and  simulation 
community's  requirements  for  environmental  data. 

The  DoD  is  also  researching  the  development  of  authoritative  models  for 
representing  military  systems  and  units  as  a  means  of  enhancing  interoperability  and  reuse. 
Efforts  are  underway  to  develop  a  joint  simulation  object  library  and  develop  common 
approaches  for  representing  threat  forces  and  systems.  Currently,  however,  there  are  no 
broadly  accepted  standards  to  represent  military  systems  and  units,  which  causes 
incompatibility  among  the  different  models. 

The  representation  of  friendly  and  threat  humans  and  groups  of  humans  under  the 
stress  of  combat  is  an  extremely  challenging  task.  The  U.S.  Army  Modular  Semi- Automated 
Forces  (ModSAF)  program  is  attempting  to  provide  a  baseline,  standardized,  modular 
software  structure  in  which  the  model  components  have  a  well-documented  and  defined 
interface.  This  program  is  seeking  to  develop  more  sophisticated,  generahzed  representations 
of  behaviors,  missions,  and  behavior  control  mechanisms.  Other  efforts  are  focused  on 
representing  command  and  control  in  entity-based  simulations,  but  there  are  few  efforts,  to 
date,  to  represent  tactical  human  behavior  authoritatively,  particularly  threat  forces,  their 
doctrine,  and  tactics. 

4.  Fielding  M&S  Systems  and  Associated  Infrastructure 

To  fully  realize  the  benefits  of  widespread  modeling  and  simulation  use,  attention 
must  be  paid  to  fielding  M&S  systems  that  are  interoperable,  reusable,  support  operational 
needs,  and  address  the  full  range  of  defense  missions.  The  fielding  must  be  in  adequate 
numbers  to  meet  DoD-wide  end  user  needs.  To  be  accepted,  modeling  and  simulation  needs 
to  have  established  credibility.  Procedures  for  verification,  validation,  and  accreditation 
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(VY&A)  need  to  better  defined  and  refined.  Furthermore,  W&A  must  begin  much  earlier 
in  model  development  and  become  a  part  of  the  M&S  life  cycle.  A  reusable  communications 
inifrastructure  with  the  capacity  to  adequately  support  modeling  and  simulation  does  not  exist 
but  efforts  are  ongoing  in  this  area.  The  DSI  has  been  implemented  to  support  geographically 
distributed  users.  Plans  also  exist  to  merge  DSI  with  the  Defense  Research  and  Engineering 
Network  (DREN)  to  enhance  capability  and  reliability.  The  long  term  objective  is  to  use 
commercial  services  and  operational  communications  capabilities  to  meet  M&S  needs,  but 
challenges  such  as  bandwidth  reduction  and  security  must  be  met  to  make  this  objective 
feasible. 

In  summary,  modeling  and  simulation  today  is  resource  intensive  and  tends  to  be 
comprised  of  narrowly  focused  systems  that  cannot  interoperate  among  user  communities, 
and  the  systems  that  do  exist  have  not  been  properly  vahdated,  verified,  and  accredited.  The 
issues  are  being  addressed  and  significant  effort  is  being  made  to  implement  DoD-wide 
architecture,  standards,  and  protocols  to  promote  interoperability  and  software  reuse.  [Ref 
13:  Chap  3] 

B.  DOD  MODELING  AND  SIMULATION  OBJECTIVES 

The  DoD  has  adopted  six  modeling  and  simulation  objectives  to  improve  its  ability 
to  more  efficiently  develop  simulation  systems  and  promote  simulator  interoperability.  The 
objectives  are  as  follows; 

•  Provide  a  common  modeling  and  simulation  architecture 

•  Provide  timely  and  authoritative  environmental  representations 

•  Provide  authoritative  representations  of  systems 

•  Provide  authoritative  representations  of  human  behavior 

•  Provide  a  modeling  and  simulation  infrastructure  to  meet  the  developer  and  end 
user  needs 

•  Share  modeling  and  simulation  benefits  with  other  government  agencies,  industry, 
and  aUied  nations  [Ref  15] 
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C.  DOD  INITIATIVES 

1.  SIMNET  and  CCTT 

In  1983,  the  U.S.  Army  and  DARPA  began  the  development  of  the  SIMNET 
training  system  to  link  various  units  over  large  scale  computer  networks  in  a  common 
synthetic  environment.  Numerous  simulators  configured  around  a  mock  up  of  the  interior 
of  an  armored  vehicle  allowed  the  participants  to  see  a  simplistic  three-dimensional 
landscape  and  other  simulated  vehicles  participating  in  the  exercise.  The  simulated  armored 
vehicles  could  maneuver  over  the  simulated  terrain  and  engage  enemy  targets  in  concert  with 
other  fiiendly  forces.  SIMNET  made  use  of  SAPOR,  both  friendly  and  enemy,  that 
performed  automatically  according  to  specified  doctrine  and  tactics  after  being  tasked  by  the 
computer  operator.  Use  of  SAPOR  added  realism  and  computer  generated  forces  without 
the  use  of  additional  simulators  that  were  normally  required  to  generate  and  control  a 
simulated  entity.  [Ref  16:  p.  10-12] 

For  those  tasks  that  are  fully  represented  in  SIMNET,  training  with  SIMNET  is  more 
effective  than  additional  field  training.  SIMNET  has  the  capability  to  train  tank  crews  in 
35%  of  the  tasks  specified  in  the  Army  Training  Evaluation  Program  and  many  studies  have 
shown  a  positive  transfer  of  tactical  training  with  SIMNET  to  field  training  [Ref  9]. 
Research  has  shown  that  platoons  trained  with  SIMNET  improved  there  scores  on 
Situational  Training  Exercises,  designed  to  evaluate  a  tank  platoon  in  both  individual  and 
collective  tasks,  by  an  average  of  13%  while  platoons  receiving  only  addition  field  training, 
improved  there  scores  by  only  6%.  Additionally,  platoons  with  more  battle  runs  on  SIMNET 
produce  higher  scores  in  competition  for  the  Canadian  Armor  Trophy.  Athough  SIMNET 
has  been  assessed  as  providing  positive  training  benefits,  those  benefits  have  never  been 
fully  quantified.  Tests  of  the  system  have  tended  to  be  subjective,  failed  to  use  control 
groups,  and  have  even  been  altered  as  a  cost  saving  measure  due  to  the  high  cost  of  testing. 
[Ref  9] 

Several  limitations  of  SIMNET,  which  included  unrealistic  behaviors  and  modeling 
only  two  ground  combat  vehicles,  led  to  the  development  of  the  Close  Combat  Tactical 
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Trainer  (CCTT).  The  design  for  CCTT  began  in  1992  as  a  simulator  for  training  armor  and 
mechanized  forces.  CCTT  is  the  first  in  a  series  of  the  envisioned  Combined  Arms  Tactical 
Trainer  (CATT)  systems  which  will  eventually  include  simulators  to  train  artillery,  aviation, 
air  defense,  and  engineer  forces. 

A  task  force  of  the  Defense  Science  Board  found  that  computer-based  simulators 
offer  the  only  practical  and  affordable  means  for  joint  commanders  and  staffs  to  exercise 
decision  making  skills,  test  war  plans,  and  train  as  a  joint  force.  Despite  this  finding  in  1988, 
uncertainty  remains  regarding  the  training  effectiveness  of  simulation  and  how  to  properly 
integrate  computer-based  simulation  and  field  training.  While  technical  tests  provide  for 
validation  and  verification  as  to  whether  CCTT  can  meet  system  requirements,  user  testing 
is  necessary  to  evaluate  the  training  effectiveness  of  CCTT  and  its  impact  on  mission 
performance.  To  accomplish  user  testing,  the  Army  plans  the  most  expensive  and 
comprehensive  test  of  a  simulation  system  to  date,  beginmng  late  in  1996.  Units  will 
undergo  pre-training  evaluations,  training  with  CCTT,  and  post-training  evaluations  that 
include  similar  training  exercises  to  evaluate  the  impact  of  CCTT  training  on  combat 
effectiveness  and  survivability.  To  determine  the  proper  mix  of  simulation  and  field 
exercises,  different  test  groups  will  receive  different  levels  of  CCTT  training,  ranging  from 
0-6  days,  mixed  with  4-10  days  field  training  for  a  total  of  ten  days  training  for  each 
group.  One  group  will  receive  no  training  as  a  control  group.  The  test  is  designed  to  compare 
the  training  effectiveness  of  different  levels  of  simulation  training  to  conventional  field 
training.  [Ref  16:  p.  14-38] 

2.  Joint  Simulation  System 

In  the  past,  each  of  the  Services  has  developed  independent  modeling  and  simulation 
systems  that  cannot  freely  interact  with  systems  developed  by  the  other  Services.  As  a  short 
term  fix,  linking  of  the  various  models  has  been  achieved  through  the  use  of  the  multi¬ 
service  Aggregate  Level  Simulation  Protocol  (ALSP).  However,  ALSP  is  a  simulation 
protocol  with  known  limitations  and  ineflSciencies  that  make  it  unsuitable  for  long  term  use 
[Ref  17], 

In  an  effort  to  better  integrate  the  range  of  missions  of  the  Armed  Forces  into  a 
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common  framework,  the  DoD  will  migrate  to  the  JSIMS  as  a  replacement  for  the  ALSP 
confederation  of  models.  The  impetus  for  JSIMS  development  is  to  eliminate  the  individual 
services  duplication  of  effort  in  modeling  and  simulation  development  and  the  resulting 
interface  difficulties,  data  sharing  problems,  and  inefficient  use  of  funds  [Ref  14].  JSIMS 
will  provide  integration  of  constructive,  live,  and  virtual  modeling  and  simulation 
capabilities,  with  C4I  folly  supported  and  interfaced  using  real-world  equipment.  The  system 
will  include  man-in-the-loop  capability  with  incorporation  of  decisions  and  simulation  at 
all  levels,  including  the  ability  to  train  dovm  to  the  individual  level.  [Ref  17] 

Funding  and  development  of  JSIMS  will  be  a  cooperative  effort  with  participation 
from  all  four  Services.  The  Marine  Corps  will  participate  in  this  effort  by  contributing  10% 
of  the  JSIMS  core  funding  with  each  of  the  other  services  contributing  30%.  Besides 
providing  funding,  the  Services  will  identify  and  nominate  the  best  service  applications  and 
algorithms  for  use  in  the  common  framework.  The  Services  will  ensure  a  common, 
economical,  and  efficient  development  process  for  their  prototypes.  The  primary 
development  goals  are  reusability,  portability  and  interoperability  of  object-based  software 
components  while  ensuring  DIS  compatibility.  All  the  Services  have  signed  a  Memorandum 
of  Agreement  outlining  their  commitment  to  JSIMS  development,  although  the  Army 
continues  to  develop  their  next  generation  model,  WARSIM  2000,  which  parallels  the 
JSIMS  effort. 

JSIMS  development  will  initially  focus  on  the  training  environment  at  the  campaign 
level.  Initial  operational  capability  for  Joint  Task  Force  level  training  is  scheduled  for  1999, 
and  by  2003,  JSIMS  is  scheduled  to  have  foil  operational  capability  for  Service  applications. 
[Ref  18] 

D.  MARINE  CORPS  MODELING  AND  SIMULATION  EFFORTS 

1.  Initial  Efforts 

The  Marine  Corps  recognizes  that  the  advancement  in  computer  and  M&S 
technologies,  such  as  DIS,  offers  significant  benefits  in  terms  of  improved  training 
effectiveness  and  cost  reduction.  The  ability  to  integrate  various  models  and  simulators,  that 
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are  geographically  distributed,  in  a  common  synthetic  environment,  offers  the  opportunity 
for  the  Marine  Corps  to  conduct  Total  Force  training  without  the  requirement  for  units  to 
leave  their  home  bases.  Commands  can  conduct  mission  planning,  rehearsal,  and  debriefs, 
and  test  operational  and  contingency  plans  in  a  synthetic  environment  with  less  cost,  time, 
and  effort.  DIS  also  provides  for  the  development  of  virtual  prototyping  of  new  systems  and 
refinement  of  their  designs  before  the  investment  of  resources  in  the  physical  prototypes. 
Similarly,  combat  developers  can  harness  DIS  to  rapidly  and  economically  explore  new 
operational  concepts  and  doctrines. 

To  promote  the  employment  of  M&S  technologies  and  to  take  advantage  of  these 
promising  benefits,  the  Marine  Corps  has  taken  several  steps.  The  most  significant  steps 
include; 

•  Establishment  of  the  Marine  Corps  Modeling  and  Simulation  Management  Office 
(MCMSMO)  as  the  Marine  Corps  focal  point  for  modeling  and  simulation 

•  Establishment  of  an  M&S  organizational  structure  that  parallels  and  interacts  with 
the  DoD  M&S  management  structure 

•  Active  participation  in  joint  M&S  initiatives 

•  Development  and  implementation  of  the  Marine  Air-Ground  Task  Force 
(MAGTF)  Tactical  Warfare  Simulator  (MTWS)  for  battle  staff  training 

•  Establishment  of  an  Advanced  Distributed  Simulation  (ADS)  demonstration  site 
at  MCAGCC  to  support  the  Marine  Corps  participation  in  the  DARPA's 
Synthetic  Theater  Of  War  (STOW)  project 

•  Establishment  of  an  ADS  demonstration  site  and  Decision  making  Support  Center 
(DMSC)  at  Quantico,  Virginia,  for  experimentation  in  the  use  of  M&S  to  support 
analysis  efforts 

•  Implementation  of  a  M&S  Master  Plan  and  a  M&S  Investment  Strategy  [Ref.  7: 
p.  1-5] 

The  following  sections  will  provide  an  overview  of  the  Marine  Corps  modeling  and 
simulation  vision  and  some  of  the  current  efforts  to  turn  that  vision  into  a  reality. 
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E.  MARINE  CORPS  MODELING  AND  SIMULATION  VISION 

1.  Overview 

The  Marine  Corps  M&S  vision  incorporates  two  orientations.  The  first  is  internal, 
focusing  on  the  use  of  M&S  within  the  Marine  Corps  to  improve  mission  performance  across 
the  Total  Force.  The  second  orientation  is  external  and  focuses  on  Marine  Corps 
participation  in  the  DoD  M&S  initiatives. 

The  Marine  Corps  will  leverage  emerging  M&S  technologies,  such  as  DIS,  to  replace 
and/or  incorporate  the  current  stand  alone  models.  The  objective  is  to  augment  traditional 
live  fire  training  at  all  levels  through  the  integration  of  combat  systems,  command  and 
control  nodes,  trainers,  logistics  and  support  systems,  and  constructive  models  and  war 
games  into  a  common  synthetic  environment.  Linking  of  the  various  individual,  crew,  and 
small  unit  simulators  into  a  common  synthetic  environment  will  support  MAGTF  Battle 
Staff  training  at  all  levels  as  well  as  joint-level  training.  The  end  result  will  permit  the  Total 
Force  to  train  as  it  fights  whether  in  garrison,  deployed  aboard  ship,  or  deployed  in  other 
forward  areas.  [Ref.  7:  p.  1-9] 

2.  Envisioned  M&S  Endstates 

To  maximize  the  full  potential  of  M&S  technology,  connectivity  will  play  a  critical 
role  in  the  realizing  the  Marine  Corps  M&S  vision.  The  various  models  and  simulators  used 
at  all  levels  of  training  require  interconnectivity  not  only  within  the  Marine  Corps,  but  in  the 
joint  environment  as  well.  Developing  this  network  of  simulators  on  a  global  scale  will  foster 
Marine  Corps  participation  in  joint  training  and  planning  exercises  and  permits  a  role  in 
development  and  evaluation  of  joint  tactics  and  doctrine. 

M&S  is  additionally  envisioned  as  pla5dng  a  critical  role  in  the  Marine  Corps  Combat 
Development  Process.  Simulated  environments  will  be  used  to  develop  concepts,  validate 
requirements,  and  conduct  course  of  action  (CO  A)  analysis  by  exercising  force  on  force  and 
warrior-in-the-loop  simulations.  Acquisition  costs  can  be  reduced  through  the  use  of  virtual 
prototyping  that  will  replace  the  need  for  expensive  physical  prototyping  to  design  and  test 
combat  system  effectiveness.  System  designs  will  be  refined  prior  to  incurring  the  cost  of 
physical  development  with  the  use  of  simulated  environments  to  evaluate  design  alternatives 
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and  analyze  the  performance  tradeoffs.  As  system  development  matures,  simulated 
environments  will  be  used  to  evaluate  and  revalidate  the  system  performance  prior  to  full- 
scale  development  and  production. 

In  an  effort  to  guide  the  attainment  of  the  Marine  Corps  M&S  vision  and  to  focus 
Marine  Corps  efforts,  a  set  of  eight  M&S  endstates  have  been  laid  out  in  the  Marine  Corps 
Modeling  and  Simulation  Master  Plan.  These  endstates  are  as  follows: 

•  Exercise  any  size  Total  Force  MAGTF  as  part  of  a  combined  or  joint  force  from 
home  bases,  aboard  ship,  or  forward  deployed  through  the  seamless  integration 
of  live,  virtual,  and  constructive  simulations 

•  Conduct  mission  planning  in  a  distributed  environment 

•  Conduct  mission  preview  and  rehearsal  on  land  or  at  sea  at  all  levels,  from  the 
individual  Marine  to  a  Marine  Expeditionary  Force  (MEF),  within  48  hours  of 
tasking 

•  Validate  Marine  Corps  requirements  and  doctrine  using  M&S  as  a  primary  tool 

•  Participate  in  the  fundamental  improvement  of  the  acquisition  process  by 
simulating  before  "we  buy,  build,  or  fight" 

•  Merge  M&S  and  command,  control,  and  communications  systems 

•  Support  every  major  weapon  system  in  the  Marine  Corps  with  a  simulator  that 
can  be  networked  into  a  common  synthetic  environment 

•  Use  M&S  as  a  primary  decision  support  tool  [Ref  7;  p.  1-12] 

F.  TEAM  TARGET  ENGAGEMENT  SYSTEM 

The  Team  Target  Engagement  System  (TTES)  is  a  Marine  Corps  sponsored 
advanced  technology  demonstration  to  develop  a  trainer  for  individuals  and  small  units.  This 
simulator  will  enable  users  to  participate  in  synthetic  force  on  force  combat  in  an  urban 
environment  and  to  engage  interactively  with  Computer  Controlled  Hostiles  (CCH).  The 
objective  is  to  provide  the  users  with  tactical  realism  and  hostile  reactivity  to  fiiendly  force 
actions  while  avoiding  the  cost  and  use  of  human  instructors  and  other  simulator  stations  to 
control  the  simulation  of  other  participants  in  the  exercise. 
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The  focus  of  CCH  development  is  to  create  hostile  entities  that  are  realistic 
participants  in  the  simulation  by  designing  them  to  move  exactly  as  humans  do  in  an  urban 
combat  environment,  suffer  wounds  when  hit,  and  "see"  users  when  they  are  visible  and 
react  accordingly.  To  accomplish  this,  the  development  must  take  into  account  the  opposing 
force  mission,  available  resources,  acceptable  losses,  characteristics  of  the  terrain  and  the 
structures,  as  well  as  the  actions  of  the  users.  To  achieve  the  objective  of  developing  entities 
that  will  operate  autonomously  without  the  control  of  an  operator,  the  individual  human 
represented  by  the  system  will  consist  of  a  physical  model  level,  a  control  level,  a  reactive 
decision  level,  and  a  reasoning  level.  [Ref  19] 

The  physical  level  will  describe  human  movement,  action  capabilities,  simple  models 
of  weapons  use,  performance  degradation  from  wounds,  and  audio  and  visual  perception. 
The  reactive  decision  level  selects  appropriate  action  for  the  immediate  situation  and  is  to 
be  designed  to  have  minimal  computation  at  this  level  so  that  the  decision  process  can  be 
repeated  frequently.  The  reactive  decision  level  will  also  be  designed  to  incorporate  planning 
capabilities  several  seconds  to  several  minutes  into  the  future  to  provide  a  more  realistic 
hostile  response  to  friendly  actions.  Finally,  the  control  level  serves  to  govern  movement  and 
continuous  low  level  processes. 

In  addition  to  developing  the  individual  human  entities,  a  major  objective  of  TTES 
is  developing  a  semantically  rich  representation  of  buildings.  In  doing  so,  it  will  allow  the 
computer  generated  forces  to  reason  about  such  issues  as  movement,  routes,  cover  and 
concealment,  and  firing  positions  by  providing  information  about  wtills,  apertures,  rooms, 
stairs,  and  other  urban  features. 

TTES  will  be  designed  to  be  DIS  compliant  so  that  these  simulators  can  interact  with 
other  simulators  such  as  LeatherNet.  However,  since  current  DIS  protocols  do  not  allow 
simulation  of  individual  combatants,  the  TTES  project  must  propose  extensions  to  the 
existing  protocols.  [Ref  19] 
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G.  LEATHERNET 

1.  Background 

The  advancement  of  computer  and  visualization  technologies,  specifically  virtual 
visualization  of  combat  scenarios,  has  led  DARPA  to  apply  these  technologies  toward  the 
development  of  training  and  scenario  analysis  tools.  To  demonstrate  the  capabilities  of  these 
tools,  DARPA  has  planned  STOW-97  as  a  proof-of-principal  demonstration  with  a  large 
scale  joint  exercise  using  live,  virtual,  and  constructive  forces.  The  Defense  Simulation 
Internet  will  be  used  to  connect  these  forces  comprising  50,000  entities.  STOW-97  will 
provide  a  synthetic  environment  in  which  human  users  can  interactively  conduct  realistic 
wartime  scenarios  involving  fiiendly  and  enemy  forces,  with  the  ability  to  review  and 
debrief  the  events,  and  conduct  "what  if  analysis.  [Ref  20:  p.  1-4] 

2.  Overview 

LeatherNet  is  a  virtual  simulation  system  that  is  being  developed  at  MCAGCC,  29 
Palms,  California,  under  the  sponsorship  of  DARPA  The  LeatherNet  project  provides 
DARPA  the  opportunity  to  develop  U.S.  Marine  Corps  Semi-Automated  Forces  (MCSAF) 
for  use  in  the  STOW-97  demonstration.  DARPA's  objective  is  to  capture  the  unique  features 
and  behaviors  of  Marine  Corps  systems.  The  objective  requires  development,  in  a  distributed 
model,  of  the  arbitration  of  battle  outcomes  at  the  entity  level  of  detail  to  include  portrayal 
of  both  individual  weapons  systems  and  individual  combatants.  Additionally,  eflForts  are 
being  taken  to  provide  a  realistic  representation  of  Marine  Corps  command  nodes,  both 
ashore  and  afloat,  which  will  portray  the  influences  of  one  command  level  over  the  actions 
of  another.  The  end  product  of  the  LeatherNet  project  is  intended  to  represent  a  MEF  in 
ground  maneuver  warfare,  to  include  amphibious  functions,  with  all  the  weapons  systems 
and  the  various  positions  in  the  MEF  organization  in  which  humans  function.  [Ref  20] 

3.  Project  Strategy 

LeatherNet  development  is  being  implemented  in  four  builds.  The  first  two  builds 
were  conducted  during  FY  1994  and  1995  and  builds  three  and  four  will  be  completed 
during  FYs  1996  and  1997  respectively.  A  brief  summarization  of  the  builds  follows: 

•  Build  1  Consists  of  developing  a  capability  to  conduct  a  "terrain  walk"  of 
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Range  400  at  MCAGCC,  29  Palms  and  the  ability  for  a  Company  Command 
Element  and  subordinate  leaders  to  conduct  mission  planmng,  briefing,  rehearsal, 
and  debrief  for  this  range  in  a  simulated  environment. 

•  Build  2  Implements  a  capability  for  Company  Command  Elements  and 
subordinate  leaders  to  conduct  a  "terrain  walk"  of  the  Delta  Corridor  at  29  Palms 
as  well  as  conduct  mission  planmng,  briefing,  rehearsal,  and  debriefing  for  this 
training  area  in  a  simulated  environment. 

•  Build  3  Implements  the  same  capabilities  as  in  Build  2  except  that  development 
is  focused  on  the  Battalion  Landing  Team  Command  Element  and  subordinate 
leaders. 

•  Build  4  Implements  the  same  capabilities  as  in  Build  2  except  that  development 
is  focused  on  the  Regimental  Landing  Team  Command  Element  and  subordinate 
leaders. 

Development  of  LeatherNet  is  dependent  upon  the  Tactical  Exercise  Evaluation  and 
Control  Group  (TEECG)  instructional  staff  at  29  Palms  to  provide  feedback  to  ensure  that 
the  existing  Range  400  and  Delta  Corridor  training  scenarios  are  accurately  represented  by 
the  system.  [Ref  20:  p.  11-16] 

4.  Benefits  of  LeatherNet 

The  envisioned  benefits  of  LeatherNet  are  to  provide  the  Marine  Corps  with  lower 
cost  and  more  effective  training.  These  benefits  will  be  derived  through  the  capabihty  of 
studying  different  tactical  alternatives  and  conducting  "what  if  analysis  without  the  use  of 
field  exercises.  LeatherNet  will  enable  units  to  conduct  large-scale  exercises  against  a  hostile 
force  without  risk  of  casualties,  expenditure  of  ammunition  and  fuel,  or  wear  and  tear  on 
equipment.  Other  benefits  of  virtual  simulation,  as  previously  discussed,  can  be  gained  from 
LeatherNet's  implementation  as  well.  These  benefits  may  be  enhanced  by  providing 
LeatherNet  capability  to  other  locations  in  addition  to  the  facility  at  MCAGCC. 
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m.  EVALUATION  OF  SIMULATION  SYSTEMS 


A.  VALIDATION  AND  VERIFICATION 

To  establish  the  credibility  of  M&S  systems,  the  DoD  is  reliant  upon  validation  and 
verification  techniques.  The  validation  process  or  the  "process  of  determining  the  degree  to 
which  a  model  is  an  accurate  representation  of  the  real  world  from  the  perspective  of  the 
intended  uses  of  the  model"  and  the  verification  process  or  the  "process  of  determining  that 
a  model's  implementation  accurately  represents  the  developers  conceptual  description  and 
specification"  have  never  been  agreed  upon  [Ref.  21].  The  techniques  and  how  they  should 
be  applied  to  establish  the  credibility  of  the  M&S  system  when  used  for  a  particular 
application  is  the  subject  of  much  debate.  Many  techniques  exist  for  the  developer  to 
establish  validation  and  verification  to  include  mathematical  proofs,  design  walk  throughs, 
and  code  verification  to  name  a  few.  However,  the  basic  questions  to  be  answered  are;  "What 
do  we  need  the  M&S  system  to  do?"  and  "How  well  can  the  M&S  system  do  them?"  The  end 
result  must  be  acceptance  by  the  end  users  of  the  M&S  system.  To  reduce  the  risk  of 
executing  a  M&S  program  that  fails  to  meet  requirements,  a  review  of  the  M&S  system  in 
operation  by  subject  matter  experts  (SME)  as  early  as  possible  in  the  development  is 
essential  [Ref  21].  Determining  whether  a  system  provides  the  required  functions  and 
whether  those  functions  accurately  model  the  real  world  are  only  some  of  the  considerations 
that  should  be  taken  into  account  in  fielding  a  military  M&S  system.  The  traming 
effectiveness  and  cost  of  the  system  are  other  prominent  considerations  that  must  be 
considered. 

B.  TRAINING  EFFECTIVENESS  ANALYSIS 

There  are  no  well  defined  methods  for  determining  military  value  of  a  training 
program  or  system,  but  military  value  is  ultimately  reflected  in  the  degree  of  combat  success 
that  any  training  program  or  system  is  responsible  for.  Unfortunately,  the  degree  to  which 
a  training  program  or  system  enables  success  in  combat  is  difficult  to  assess.  Military  value 
can  be  empirically  determined  only  in  combat  and  it  is  obviously  impractical  and  unwise  to 


25 


wait  for  war  to  determine  the  eflfectiveness  of  a  training  system.  [Ref  22:  p.  7] 

As  a  more  practical  approach  to  assess  training  effectiveness,  measures  of 
effectiveness  (MOE)  are  used  as  an  alternative  to  actual  combat  to  predict  combat  success. 
MOE  are  equivalent  to  dependent  variables  that  are  used  to  assess  the  impact  of  an 
experimental  treatment  condition  [Ref.  22;  p.  7].  MOE  can  be  defined  as  "tools  that  assist 
in  discriminating  among  a  number  of  alternatives."  They  show  how  the  alternatives  compare 
in  meeting  functional  objectives  and  mission  needs.  [Ref  23:  p.  8-7] 

In  selecting  MOE,  a  distinction  can  be  made  between  objective  and  subjective  data. 
Objective  data  are  based  on  observable  events  whose  occurrence  is  not  usually  the  subject 
of  dispute.  Examples  include  a  student's  test  score  or  the  number  of  hits  scored  on  a  gunnery 
range.  Subjective  data  are  based  on  the  opinions  and  judgements  of  a  SME.  Subjective  data 
is  necessary  to  capture  intangible  information  such  as  leadership,  decisiveness,  morale,  and 
motivation.  While  the  SME  evaluation  may  carry  great  weight,  it  may  be  neither  accurate 
nor  valid.  [Ref:  22:  p.  12-14] 

Selection  of  training  MOE  is  dependent  upon  the  tasks  necessary  to  accomphsh  the 
training,  the  conditions  under  which  the  task  will  be  conducted,  and  the  standards  for  the 
tasks.  Evaluation  of  collective  training  is  much  more  dfficult  and  subjective  than  evaluation 
of  individual  training.  The  problem  is  further  compounded  when  attempting  to  evaluate 
simulators.  The  Government  Accounting  Office  (GAO)  has  concluded  that  despite  the 
Army's  strategy  to  increase  the  use  of  simulation,  the  strategy  lacks  detailed  guidance  for 
commanders  to  make  effective  use  of  simulators.  The  major  downfall  is  the  lack  of  linkage 
between  the  simulation  and  the  tasks  the  unit  can  expect  to  perform  in  wartime.  The  lack  of 
guidance  has  resulted  in  the  less  than  optimal  use  of  simulators.  That  being  the  case,  the  true 
training  value  may  not  be  realized  nor  evaluated.  [Ref  24:  p.  2] 

Deciding  how  to  measure  the  training  effectiveness  of  a  training  program  or  system 
is  neither  obvious  nor  simphstic.  The  training  environment  must  be  considered  to  determine 
whether  the  training  is  structured  or  unstructured.  Schools  tend  to  offer  structured  training 
and  be  more  objective  in  nature.  Unit  training  tends  to  be  unstructured  and  evaluation  is 
more  difficult  and  subjective  due  to  reliance  upon  the  opinion  and  judgement  of  SME. 


26 


Further  difficulty  arises  from  the  lack  of  understanding  of  the  important  dimensions  of 
collective  training  and  evaluation.  Even  developing  the  task  list  to  determine  the  MOE  is 
complicated  by  the  fact  that  the  tasks  tend  to  exist  in  hierarchies,  and  separate  task  lists  can 
be  developed  at  each  level  in  the  hierarchy.  As  the  number  of  levels  increase,  tasks  are  added 
to  each  level  for  possible  interaction  between  that  level  and  other  levels.  A  comprehensive 
task  list  would  be  comprised  of  all  the  tasks  at  each  level  and  all  the  tasks  involving 
interaction  among  levels.  No  standard  method  of  developing  collective  MOE  exists.  Many 
researchers  stress  the  importance  of  relying  upon  SME's  to  assist  in  developing  MOE.  SME's 
enable  researchers  to  identify  indicators  of  success,  interpret  performance,  and  to  ensure  all 
relevant  tasks  are  measured.  While  the  SME's  know  best  how  to  assess  the  complex  training 
environment,  objectivity  may  be  lost  as  this  method  is  reliant  upon  opinion  and  judgement, 
[Ref  22:  p.  10-18] 

In  using  MOE  certain  assumptions  must  be  made.  In  order  to  conduct  a  training 
effectiveness  analysis  of  one  training  method  over  another,  a  MOE  that  predicts  combat 
success  must  be  selected.  The  MOE  might  be  student  grades  or  an  evaluator's  rating.  To  use 
these  MOE,  one  must  make  a  transfer  assumption  that  student  grades  or  an  evaluation  rating 
predict  combat  success.  Such  an  assumption  implies  a  series  of  assumptions  that  school  or 
training  performance  influences  training  readiness  which  influences  combat  readiness.  While 
research  has  provided  data  supporting  the  linkage  between  school  performance,  field 
performance,  and  combat  readiness  levels,  these  linkages  remain  largely  unconfirmed.  [Ref 
22;  p.  13] 

C.  COST  EFFECTIVENESS  ANALYSIS 

Cost  effectiveness  analysis  (CEA)  is  a  method  used  by  the  DoD  to  make  decisions 
in  selecting  alternative  courses  of  action  when  the  results  effect  military  performance.  While 
the  inputs  can  be  measured  on  a  cost  basis,  the  outputs  or  military  value,  cannot  be  measured 
in  monetary  terms.  CEA  is  similar  to  cost  benefit  analysis  in  respect  to  taking  into  account 
all  of  the  associated  costs  of  a  project  throughout  a  projected  life  cycle.  However,  CEA  takes 
into  account  military  value  which  cannot  be  defined  in  the  same  terms  as  cost  due  to  a  lack 
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of  market  value  for  increased  readiness,  improved  training,  and  improved  weapon  systems. 
Since  cost  and  military  value  have  different  units  of  measurement,  selection  of  alternatives 
cannot  be  made  on  a  cost  basis  alone.  Orlansky  developed  a  decision  logic  diagram  for 
evaluating  the  cost  and  effectiveness  of  two  training  programs  or  systems.  The  diagram  is 
shown  below: 


EFFECTIVENESS 

LESS  SAME  MORE 


LESS 

9 

• 

+ 

+ 

COST 

SAME 

- 

9 

• 

+ 

MORE 

- 

- 

9 

• 

Table  1.  Cost  and  Effectiveness  Decision  Diagram  [Ref.  1] 

Using  the  diagram  in  Table  1,  if  an  alternative  is  as  effective  and  costs  less  than 
another,  it  should  be  adopted.  Likewise,  if  an  alternative  is  more  effective  and  cost  the  same 
or  less  than  another  alternative  it  should  be  adopted.  An  alternative  should  be  rejected  if  it 
is  less  effective  and  cost  the  same  or  more  than  the  other  alternative  or  the  alternative  is  as 
effective  and  cost  more.  Finally,  in  those  situations  where  the  alternatives  exhibit  less 
effectiveness  and  less  cost,  equal  effectiveness  and  equal  cost,  or  greater  effectiveness  and 
greater  cost,  no  logical  preference  can  be  made.  [Ref.  1 :  p.  111-2] 

While  Orlansky  recommends  acquiring  simulators  that  are  equally  effective  and  cost 
less  than  the  actual  equipment,  he  suggests  that  the  military  should  strive  to  acquire 
simulators  that  provide  increased  effectiveness  at  the  same  or  lesser  cost.  He  believes  that 
the  mihtary  should  take  the  same  approach  to  acquiring  simulators  as  it  does  to  acqiuring 
weapon  systems  with  the  focus  on  increasing  effectiveness.  Costs  should  be  a  secondary 
consideration.  The  end  result  should  be  the  optimum  combination  of  simulators  and  actual 
equipment  that  provide  the  most  effective  training  at  the  least  overall  cost.  [Ref  1 :  p.  III-l  1] 
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In  a  study  by  the  M&S  Benefits  Task  Force,  DMSO  reports  the  training  applications 
of  M&S  for  individual  skills  training  are  wide  spread  and  the  training  benefits  are  positive 
when  simulation  is  properly  mixed  with  training  on  real  equipment.  Analyst  have  well- 
established  theories  and  experimental  methods  for  conducting  analysis  at  this  level.  The 
same  cannot  be  said  for  unit  training,  particularly  in  the  higher  echelons.  The  high  cost  of 
repeated  large  scale  exercises  and  the  difficulty  of  conducting  the  controlled  experiments 
necessary  preclude  the  collection  of  meaningful  data.  Multi-million  dollar  savings  are 
reported,  however,  when  comparing  computer  simulated  command  post  exercises  with 
conventional  field  training  exercises.  [Ref  9:  p.  2] 
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IV.  METHODOLOGY 


A.  BACKGROUND 

1.  Purpose 

The  purpose  of  this  research  is  to  evaluate  the  perceived  ability  of  LeatherNet  to  meet 
the  mission  planning,  and  to  some  extent,  the  mission  preview  objectives  of  the  Marine 
Corps  Modeling  and  Simulation  endstates  as  described  in  the  Marine  Corps  Modeling  and 
Simulation  Master  Plan.  Secondly,  this  research  will  determine  if  a  correlation  exists 
between  the  exposure  to  conducting  mission  planning  and  briefing  with  LeatherNet,  and 
subsequent  user  performance  in  conducting  live  fire  training. 

2.  Test  Constraints 

User  testing  and  evaluation  of  LeatherNet  has  been  constrained  by  several  factors. 
All  aspects  of  LeatherNet  could  not  be  evaluated  since  the  system  remains  under 
development.  The  number  of  test  participants  has  been  limited  by  the  number  of  units 
participating  in  the  Combined  Arms  Exercise  (CAX)  Program,  which  are  the  units 
designated  by  the  Commanding  General  of  MCAGCC  to  conduct  user  testing  of  LeatherNet. 
The  CAX  program  is  an  ongoing  Marine  Corps  program  to  train  Marine  infantry  battalions 
in  all  aspects  of  fire  support  coordination  in  a  live  fire  and  maneuver  environment.  CAX 
training  cycles  last  approximately  30  days  and  are  comprised  of  intense  field  training  in  fire 
support  techniques  and  tactics  at  all  levels  of  command.  The  training  schedule  allows  units 
little  or  no  opportunity  to  conduct  training  other  than  that  training  related  directly  to  the 
CAX  program.  Using  other,  less  encumbered,  units  to  conduct  the  testing  is  not  practical  due 
to  the  cost  of  transportation  to  the  remote  test  site  and  the  cost  of  the  training  on  Range  400 
in  terms  of  ammunition  expenditure.  During  the  period  fi’om  April  to  August  1996,  a  total 
of  eight  groups  were  exposed  to  the  system.  The  amount  of  time  the  users  were  exposed  to 
the  system  was  constrained  by  the  demanding  CAX  training  schedule  that  limited  their 
exposure  to  three  to  four  hours  between  field  exercises.  The  time  constraints  further  limited 
the  functions  which  could  be  demonstrated,  eliminating  those  which  were  time  intensive. 
Some  functions  were  not  demonstrated  due  to  incorrect  behaviors  and  inaccurate 
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representation  of  certain  entities  at  this  stage  of  development.  For  example,  hostile  infantry 
would  stand  on  the  trench  lines,  rather  than  in  them,  rendering  them  easier  to  "kill."  In  these 
instances,  the  TEECG  personnel,  who  are  responsible  for  conducting  the  CAX  program,  felt 
the  system  provided  "negative  training"  due  to  incorrect  behavior  of  these  entities.  The 
TEECG  thought  such  behavior  gives  a  false  impression  of  the  difficulty  in  overcoming 
fortified  defenses  and  decided  against  testing  the  system  using  the  fiill  wargaming 
capabilities  until  the  represented  behaviors  become  more  realistic  with  further  development. 

3.  Test  Environment 

0.  User  Testing  Participants 

The  officer  responsible  for  coordinating  and  conducting  user  testing  of 
LeatherNet  is  Major  Daniel  Newell,  the  TEECG  LeatherNet  Representative.  Major  Newell 
ensures  that  two  of  the  three  rifle  companies  from  each  infantry  battalion  participating  in  the 
CAX  program  receives  its  Range  400  exercise  brief  using  LeatherNet.  The  third  company, 
which  is  the  control  group,  is  not  exposed  to  LeatherNet  and  receives  its  brief  through  the 
use  of  a  map  study.  The  test  participants  typically  consist  of  an  infantry  company's 
Command  Element  and  the  Platoon  Leaders.  The  companys  have  the  option  of  including 
other  individuals  in  the  briefs  such  as  their  Forward  Observers  and  Machine  Gun  Section 
Leaders  at  their  own  discretion.  A  total  of  31  participants  were  surveyed. 

The  participants  come  from  various  Marine  Corps  bases  around  the  U.S.  and 
include  Marines  from  both  reserve  and  active  duty  units.  The  participants  had  varying 
degrees  of  experience  in  their  Military  Occupational  Specialty,  to  include  prior  exercises  on 
Range  400,  but  none  had  used  LeatherNet  prior  to  the  testing. 

b.  Functions  Evaluated 

As  previously  noted,  due  to  the  maturity  of  LeatherNet  development,  time 
constraints,  and  the  willingness  of  test  personnel  to  test  the  system,  only  the  mission 
planning  and  mission  briefing  functions  of  the  Marine  Corps  objectives  for  LeatherNet  are 
evaluated.  The  system,  while  mature  enough  to  conduct  a  mission  rehearsal  against  an 
interactive  hostile  force,  and  perform  "what  if  COA  analysis,  was  not  used  for  these 
purposes  during  the  course  of  this  research.  The  LeatherNet  mission  planning  and  briefing 
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evaluation  focused  on  using  the  system  to  conduct  an  estimate  of  the  situation  to  include 
taking  into  consideration  such  elements  as  mission,  enemy,  troops  and  fire  support 
available,  time,  space,  and  logistics.  The  test  participants  were  led  through  the  estimate  of 
the  situation  analysis  by  Major  Newell  as  part  of  the  TEECG's  Range  400  exercise  brief. 
During  the  brief,  the  participants  were  allowed  to  use  LeatherNet  to  "navigate"  the  Range 
400  terrain  to  conduct  a  reconnaissance  from  both  the  fiiendly  and  enemy  perspectives.  In 
doing  so,  the  participants  were  able  to  assess  their  plan's  vulnerability  firom  potential  enemy 
positions  and  weapons  systems  along  their  planned  maneuver  routes  by  using  LeatherNet's 
display  of  weapons  range  fans  and  deadspace.  Likewise,  using  the  same  functions,  they  were 
able  to  evaluate  the  effective  target  coverage  of  fiiendly  weapon  systems  from  potential 
fiiendly  firing  positions.  Additionally,  the  "terrain  walk"  of  Range  400  enabled  the 
participants  to  "see"  the  terrain,  the  potential  enemy  positions,  and  take  into  account  time 
and  space  consideration  for  their  fire  support  and  maneuver  elements. 

User  surveys,  to  measure  user  opinion  of  LeatherNet,  were  conducted 
immediately  after  the  completion  of  the  Range  400  field  exercise  to  ensure  minimal 
information  loss  by  the  participants  due  to  a  time  lag  between  the  event  and  the  survey.  The 
user  survey  can  be  found  in  Appendix  A. 

4.  Range  Performance  Evaluation 

The  analysis  conducted  in  this  study  is  based  on  a  comparative  evaluation  between 
the  experimental  groups,  those  units  given  a  mission  brief  using  LeatherNet  prior  to 
conducting  the  Range  400  exercise,  and  the  control  groups,  those  units  given  a  mission  brief 
using  only  a  map  and  aerial  photo  study.  The  comparison  of  the  two  groups  is  made  by 
comparing  the  percentage  of  the  31  training  objectives  for  Range  400,  as  developed  by  the 
TEECG  staff,  that  are  successfully  accomplished  by  each  group.  The  Range  400  evaluation 
worksheet  is  included  in  Appendix  B.  Assigning  scores  to  the  evaluation  worksheet  was 
performed  by  a  single  SME,  Major  Newell  of  the  TEECG.  The  best  alternative  training 
method  is  determined  by  the  alternative  with  the  highest  average  percentage  of  training 
objectives  successfully  accomplished.  In  some  instances,  not  all  31  objectives  could  be 
measured  due  to  the  absence  of  certain  sub-units  conducting  the  exercise.  For  example,  the 
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absence  of  Dragon  missile  teams  precludes  their  evaluation.  In  these  instances,  the  total 
score  for  the  unit  was  based  on  the  percentage  of  remaining  objective  successfully 
accomplished.  The  number  of  training  objectives  evaluated  for  the  eight  experimental  and 
four  control  groups  ranged  jfrom  28  to  3 1 . 

If  the  experimental  groups  demonstrate  a  higher  achievement  of  training  objectives 
to  a  statistically  significant  degree,  it  can  be  assumed  a  positive  transfer  of  military  value  has 
occurred.  However,  nothing  can  be  inferred  about  the  transfer  as  a  function  of  the  amount 
of  exposure  to  LeatherNet.  To  do  so  would  require  several  different  units  to  experience 
LeatherNet  exposures  of  different  time  lengths  and  subsequent  evaluation  on  the  range.  [Ref 
22;  p.  30] 

5.  Threats  to  Validity 

Validity  refers  to  the  extent  to  which  the  user  surveys  and  performance  evaluations 
actually  measure  the  true  benefits  the  users  derive  from  exposure  to  LeatherNet.  To  answer 
the  basic  research  questions,  numerous  investigate  questions  were  considered  by  the  author. 
These  questions  were  pretested  to  eliminate  ambiguous  questions  and  to  detect  other 
weaknesses  in  the  survey  instrument,  and  refined  into  the  questions  found  on  the  survey 
included  in  Appendix  A.  To  enhance  content  validity,  a  review  of  the  user  survey  by  SME 
within  TEECG  was  conducted  to  ensure  adequate  coverage  of  the  questions  under  study.  The 
performance  evaluation  forms  for  Range  400  used  in  this  study  were  developed  by  TEECG 
prior  to  this  study  and  are  considered  to  be  adequate  to  measure  performance  since  they  have 
been  developed  by  the  SME  responsible  for  evaluating  Range  400  exercises  as  part  of  their 
official  duties. 

The  study  consists  of  a  post-test  only  control  group  experiment.  No  performance 
pretest  was  conducted  with  either  group.  The  control  and  experimental  groups,  consisting 
of  infantry  company  command  elements  and  subordinate  platoon  and  section  leaders,  were 
determined  by  random  selection.  In  a  post-test  only  control  group  study,  normally  the 
internal  threats  to  validity  from  history,  maturation,  selection,  and  statistical  regression  are 
adequately  controlled  by  random  assignment  of  groups.  [Ref  25;  p.  359-364] 

Despite  efforts  to  promote  internal  validity,  the  test  environment  experienced  during 
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this  research  presented  certain  threats  to  the  validity  of  the  experiment  used  to  measure  the 
benefits  of  LeatherNet.  Ideally,  neither  group  would  be  exposed  to  any  information  source 
concerning  the  Range  400  exercise  than  that  which  is  normally  available  to  company-sized 
units  prior  to  an  attack  upon  enemy  held  territory.  Typically,  this  information  would  consist 
only  of  intelligence  summaries  fi’om  higher  headquarters,  aerial  photos,  and  map  studies. 
Both  groups  received  all  of  these  types  of  information  with  the  experimental  group  receiving 
additional  information  available  through  the  use  of  LeatherNet.  There  were,  however,  no 
effective  means  to  control  access  to  information  sources  not  normally  available  in  a  hostile 
environment.  Prior  to  conducting  the  exercise,  units  could,  and  in  many  instances  did,  gain 
additional  information  through  physical  reconnaissance  of  Range  400,  observing  other  units 
conducting  the  Range  400  exercise,  and  receiving  advice  from  individuals  and  units  that  had 
previously  run  the  range.  If  the  control  units  chose  to  exploit  these  sources  of  information, 
it  could  negate  the  ability  to  measure  the  benefits  of  using  LeatherNet  since  this  information 
is  similar  to  the  information  LeatherNet  provides. 

The  manner  in  which  the  TEECG  conducts  its  mission  briefs  for  all  units  prior  to  the 
exercise  effectively  reduces  the  possibility  of  critical  errors  by  the  exercising  units,  and 
therefore,  tends  to  diminish  disparity  that  might  normally  be  observed  between  the  exercise 
and  control  groups.  The  TEECG's  purpose  is  to  assist  and  evaluate  unit  training  and  their 
efforts  are  often  focused  on  helping  units  train  correctly,  rather  than  allowing  units  to  fail 
due  to  lack  of  experience  and  simply  giving  them  an  evaluation  of  their  shortcomings.  As 
such,  training  exercise  briefs  often  includes  instruction  that  is  prescriptive  in  nature,  causing 
units  to  execute  very  similar  plans  with  little  variation  due  to  the  suggestive  remarks.  For 
example,  supporting  weapons  such  as  mortars  and  heavy  machine  guns  are  employed  from 
essentially  the  same  positions  on  Range  400  by  nearly  every  unit.  The  reasons  for  this  are 
not  only  due  to  suggestive  remarks,  but  suggestive  names  for  terrain  features  such  as 
Machine  Gun  Hill.  Specific  instructions  for  weapons  employment  were  also  given  that  left 
little  room  for  variance,  due  to  training  safety  considerations.  Under  these  circumstances, 
the  ability  of  LeatherNet  to  influence  performance  tends  to  be  reduced. 

Other  influences  upon  unit  performance  evaluation  that  could  not  be  controlled 
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include  the  amount  of  ammunition  available  to  the  units  and  the  personnel  used  to  conduct 
the  evaluation.  When  ammunition  availability  is  limited,  planning  is  essential  to  performance 
to  ensure  that  ammunition  remains  available  in  adequate  amounts  at  critical  moments  of  the 
exercise  to  ensure  fire  support  is  available.  However,  the  author  observed  some  units 
conducting  the  range  with  up  to  three  times  the  normal  allotment  of  some  ammunition  types. 
Such  abundant  supplies  of  ammunition  give  the  units  a  performance  edge  that  enables  them 
overcome  the  lack  of  information  and  planning  prior  to  the  exercise.  An  internal  vaUdity 
threat  also  exists  in  the  fact  that  the  performance  evaluations  were  conducted  by  a  single 
individual.  The  evaluator's  boredom,  experience,  anticipation,  bias,  and  fatigue  all  have  the 
potential  to  influence  results. 

It  is  acknowledged  that  it  is  difficult  to  conduct  an  experimental  test  in  an  operational 
setting.  The  problems  include  difficulty  in  establishing  experimental  control  in  manipulating 
the  events  for  experimental  purposes.  In  order  to  achieve  this  objective,  the  cost  becomes 
prohibitive  as  it  requires  using  units  that  are  solely  dedicated  to  the  test  process,  as  well  as 
the  expenditure  of  large  amounts  of  ammunition  to  replicate  a  combat  scenario.  To  do  this 
for  the  single  purpose  of  testing  LeatherNet  was  considered  impractical  for  a  system  still 
under  development. 

B.  DATA  COLLECTION 

1.  Questionnaire  Development 

A  user  survey  questionnaire  was  developed  to  measure  the  user's  perception  of  the 
value  of  the  various  aspects  of  LeatherNet  as  a  mission  planning  and  mission  briefing  tool. 
Many  of  the  specific  questions  were  developed  from  a  questionnaire  previously  used  by 
Major  Newell  to  conduct  user  opinion  surveys  regarding  the  use  of  LeatherNet  to  conduct 
an  estimate  of  the  situation.  This  questionnaire  was  revised,  reformatted,  and  expanded  to 
cover  other  aspects  of  this  research.  The  user  survey  was  pretested  at  the  Naval 
Postgraduate  School  using  Marine  personnel,  several  of  whom  had  previous  Range  400 
experience,  and  with  the  TEECG  personnel.  Pretesting  was  necessary  to  ensure  the  questions 
were  understandable,  unambiguous,  and  in  an  easily  readable  format.  Numerous  revisions 
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were  necessary  before  the  final  version  could  be  made. 

2.  Types  of  Data  Collected 

Questions  1  -  34,  39,  and  41-44  provide  ordinal  data  using  five  point  Likert  scales 
to  record  the  respondent's  answers.  The  respondents  are  asked  to  rank  their  degree  of 
agreement  with  the  question  statements,  ranking  fi'om  strongly  disagree  to  strongly  agree. 
Question  38  also  provides  ordinal  data  but  does  so  by  asking  the  respondents  to  rank,  by 
order  of  importance,  various  potential  capabilities  of  LeatherNet.  The  rem^der  of  the 
survey,  questions  35  -37  and  40,  provide  nominal  data.  These  questions  require  the 
respondents  to  list  LeatherNet  characteristics  that  they  found  most  useful,  least  useful  or 
thought  should  be  developed  for  the  system.  Nominal  data  was  derived  by  grouping  the 
characteristics  that  the  respondents  listed  by  category.  The  following  subsections  describe 
the  purpose  for  the  specific  questions  used  in  the  user  survey. 

3.  Determine  the  User's  Experience  Level  with  Range  400. 

Questions  1  -  8  ask  the  user  to  indicate  his  professional  experience  level,  the  tactical 
proficiency  of  his  unit,  and  his  familiarity  with  Range  400.  The  purpose  for  these  questions 
is  to  test  the  hypothesis  that  LeatherNet  is  most  useful  as  a  mission  planning  tool  to  those 
personnel  whom  are  the  least  experienced  and  familiar  with  Range  400.  In  order  to  obtain 
an  overall  rating  for  an  individual  respondent's  experience  and  familiarity,  the  various 
questions  are  weighted  according  to  their  relevance  to  professional  experience  and  Range 
400  familiarity.  The  weights  were  assigned  through  the  use  of  the  author’s  and  the  TEECG 
SME’s  judgement  after  using  pairwise  comparisons  between  the  criteria  to  determine  their 
relative  importance  as  it  relates  to  experience  To  obtain  the  overall  experience  and 
familiarity  of  the  user,  the  scores  of  all  the  questions  were  summed  to  provide  an  aggregate 
experience  and  familiarity  score.  The  possible  scoring  range  for  experience  and  familiarity 
ranges  fi’om  1,  for  least  experienced;  to  100,  for  most  experienced.  Table  2,  on  the  following 
page,  indicates  how  questions  1-8  of  the  user  survey  were  weighted  according  to  the 
respondent's  answers. 
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Question  1 

>12 

9-12 

6-9 

3-6 

<3 

Scores 

5 

4 

3 

2 

1 

Question  2 

>24 

18-24 

12-18 

6-12 

<6 

Scores 

10 

8 

6 

3 

0 

Question  3 

Yes 

No 

Scores 

20 

0 

Question  4 

>10 

7-10 

4-6 

n 

<1 

Scores 

15 

12 

8 

■■ 

0 

Question  5 

Far  Above 

Above  Ave. 

Average 

Below  Ave. 

Far  Below 

Scores 

15 

12 

8 

4 

0 

Question  6 

Yes 

No 

Scores 

15 

0 

Question  7 

Yes 

No 

Scores 

10 

0 

Question  8 

Yes 

No 

Scores 

10 

0 

Table  2.  Determination  of  User  Experience  Through  Assignment  of  Experience 
Scores  (Bottom  number  in  each  cell)  to  Possible  Responses  (Top  portion  of  each  ceil) 

to  Survey  Questions  1-8 


The  top  portion  of  each  cell  represents  a  possible  response  to  each  question  and  the 
bottom  number  reflects  the  experience  score  assigned  to  the  respondent  if  he  selected  that 
particular  answer.  A  sum  of  all  responses  to  questions  1-8  is  used  to  assign  an  overall 
experience  score  for  each  respondent. 

Admittedly,  these  questions  are  not  all  inclusive  in  determining  experience  and 
familiarity  but  represent  those  factors  which  SME  feel  have  the  greatest  degree  of  influence 
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on  experience  as  it  relates  to  performance  on  Range  400.  Additionally,  while  this  method 
of  assigning  weights  to  the  experience  and  familiarity  criteria  is  subjective,  it  is  used  due  to 
a  lack  of  a  suitable  alternative. 

4.  Determine  the  Ability  of  LeatherNet  to  Conduct  an  Estimate  of  the 
Situation 

Questions  9  -23  ask  the  respondent  his  opinion  of  the  degree  to  which  LeatherNet 
helped  his  understanding  of  the  enemy  situation,  terrain,  fire  support,  time  and  space  issues, 
and  logistical  considerations. 

5.  Determine  the  Realism  of  the  LeatherNet  Terrain 

The  users  are  asked  in  question  24  to  indicate  how  closely  the  LeatherNet  terrain 
represents  the  real  terrain  on  Range  400. 

6.  Determine  How  the  Information  Provided  by  LeatherNet  is  Used  by  the 
Test  Participants 

Questions  25,  26,  41,  and  42  ask  if  critical  information  was  provided  by  LeatherNet 
and  if  that  information  influenced  the  respondent's  plan  of  attack.  Questions  41  and  42  were 
added  to  the  end  of  the  survey  because  they  might  be  considered  "hot  button"  issues.  The 
questions  ask  the  respondent  to  indicate  if  they  failed  to  change  their  plan  of  attack  despite 
the  fact  that  LeatherNet  provided  information  indicating  they  should  do  so.  The  purpose  for 
the  questions  is  to  gauge  the  level  of  trust  users  place  in  the  information  LeatherNet 
provides. 

7.  Determine  the  Potential  of  LeatherNet  to  be  a  Mission  Planning  and 
Rehearsal  Tool 

The  respondents  are  asked,  in  questions  27  -  30,  to  indicate  the  degree  to  which 
LeatherNet  demonstrates  the  potential  to  be  an  effective  training  tool  for  Marines  to  evaluate 
tactical  alternatives,  conduct  mission  planning,  and  mission  rehearsal. 

8.  Determine  the  Value  of  LeatherNet  Under  Different  Circumstances  of 
Use 

The  respondents  are  asked  their  opinion  of  LeatherNet's  value,  in  questions  31-34, 
if  they  had  no  other  source  of  information  regarding  Range  400,  if  they  had  the  opportunity 
to  expose  more  of  their  own  Marines  to  the  system,  and  if  they  were  allowed  to  use 
LeatherNet  at  their  home  base  prior  to  CAX. 


39 


9.  Determine  the  Most  and  Least  Useful  Features  of  LeatherNet  and  What 
Features  and  Changes  can  be  Implemented  to  Make  LeatherNet  More 
Useful 

The  respondents  are  asked  in  questions  35  -37  and  40  to  provide  written  comments 
concerning  what  features  they  felt  were  the  most  and  least  useful,  what  features  should  be 
added,  and  what  changes  should  be  made  to  LeatherNet  to  make  the  system  more  useful. 

10.  Determine  the  Ranking,  in  Order  of  Importance,  of  the  Perceived 
Abilities  of  LeatherNet  to  Overcome  the  Limitations  of  Live  Fire 
Training 

Question  38  asks  the  respondent  to  rank,  in  order  of  importance,  eight  perceived 
abilities  of  LeatherNet  to  overcome  the  limitations  of  live  fire  training.  The  perceived  ability 
of  LeatherNet  to  overcome  the  limitations  of  live  fire  training  were  determined  through 
previous  research  on  the  system.  The  perceived  limitations  of  live  fire  training  include  lack 
of  realistic  and  interactive  enemy  forces,  limited  feedback  on  weapons  employment,  safety 
constraints,  lack  of  adequate  range  facilities  and  limited  ability  to  reconstruct  exercises.  [Ref. 
26;  p.  48] 

11.  Determine  the  Respondents  Attitude  Toward  LeatherNet 

Questions  43  and  44  ask  the  respondent  to  indicate  his  attitude  toward  LeatherNet 
prior  to  using  the  system  and  whether  his  attitude  is  more  positive  after  exposure.  The 
purpose  of  these  questions  is  to  determine  if  there  is  an  existing  bias  towards  using 
simulators  for  training  and  to  determine  LeatherNet's  degree  of  acceptance  among  its  users. 
C.  ANALYSIS  PLAN 

1.  Survey  Data 

The  data  collected  through  user  surveys  in  most  cases  are  ordinal  data.  The  questions 
in  the  survey  rely  upon  opinion  scales  to  register  the  participants  level  of  agreement  to 
various  statements.  To  determine  the  central  tendency  of  all  respondents  the  median  value 
is  used  and  is  presented  in  bar  graphs  to  show  the  various  levels  of  agreement  with  the 
statements  in  the  questionnaire.  The  median  is  the  most  appropriate  measure  to  determine 
the  center  for  ordinal  data  and  provides  resistance  to  extreme  scores.  In  some  cases,  the 
percentages  of  respondents  giving  particular  answers  is  given  to  further  clarify  the 
distribution  of  responses  and  levels  of  agreement. 
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The  level  of  experience,  as  calculated  from  survey  questions  1  -  8,  is  further  analyzed 
through  the  use  of  Speannan's  Rho,  to  determine  if  there  is  a  correlation  between  the  level 
of  experience  of  a  user  and  the  perceived  benefits  of  using  LeatherNet.  Spearman’s  Rho  is 
used  to  test  the  null  hypothesis,  that  there  is  no  difference  between  a  participant’s  experience 
and  the  degree  to  which  LeatherNet  provides  critical  information  and  influences  the  user’s 
plan  of  attack  [Ref  25;  p.  507-509],  The  user's  experience  score  is  correlated  with  his 
responses  to  questions  25  and  26  to  make  this  determination. 

2.  User  Performance  Data 

Performance  scores  for  the  units  using  LeatherNet  and  the  control  units  are  compared 
to  determine  if  there  is  a  quantitative  performance  differential  between  the  units  using 
LeatherNet  to  conduct  mission  planning  and  the  control  groups,  units  vrith  no  LeatherNet 
exposure.  The  comparison  of  the  two  groups  is  made  by  comparing  the  percentage  of  the 
3 1  training  objectives  for  Range  400,  as  developed  by  the  TEECG  staff,  that  are  successfully 
accomplished  by  each  group.  The  evaluation  sheet  for  Range  400  is  included  in  Appendix 
B. 

Due  to  the  small  sample  sizes  involved,  an  n  of  four  for  the  control  group  and  an  n 
of  eight  for  the  experimental  group,  the  t  test  is  used  to  test  for  a  statistically  significant 
difference  in  the  mean  percentage  of  training  objectives  successfully  completed  by  each 
group.  The  t  test  is  chosen  because  the  data,  or  performance  scores,  are  interval  level  data 
and  the  samples  are  independent.  Use  of  the  t  test  assumes  normally  distributed  populations 
and  equal  population  variances  [Ref  25;  p.  449-451].  This  test  is  designed  to  test  the  null 
hypothesis,  that  there  is  no  difference  in  performance  on  the  Range  400  exercise  for  the  units 
using  LeatherNet  as  compared  to  the  units  without  LeatherNet  exposure.  The  alternative 
hypothesis  is  that  using  LeatherNet  does  make  a  difference  in  performance. 
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V.  DATA  ANALYSIS 


A.  USER  PERCEPTIONS 

1.  Description  of  the  Chart  Data 

The  bar  charts  used  in  this  chapter  use  numbers  1  -  5  to  represent  the  user's  level  of 
agreement  with  the  survey  questions.  The  numbers  translate  as  follows:  1  -  Strongly 
Disagree,  2  -  Disagree,  3  -  Neither  Agree  nor  Disagree,  4  -  Agree,  5  -  Strongly  Agree. 
Some  charts  do  not  display  bars  for  all  five  numbers  which  indicates  that  no  test  participants 
selected  the  response  category  represented  by  the  missing  number. 

2.  Assessment  of  LeatherNet  as  a  Tool  to  Conduct  an  Estimate  of  the 
Situation. 

The  user's  perception  of  using  LeatherNet  as  a  tool  to  conduct  an  estimate  of  the 
situation  is  generally  positive.  The  median  value  of  all  responses  to  survey  questions  9-23 
are  displayed  in  Chart  1. 


Chart  1.  Assessment  of  LeatherNet  for  Conducting  an  Estimate  of  the 
Situation 

Of  all  the  elements  of  conducting  an  estimate  of  the  situation,  the  users  agreed  most 
strongly  that  LeatherNet  improves  their  understanding  of  the  enemies  fields  of  fire  (question 
10)  with  67%  of  all  respondents  answering  “strongly  agree.”  The  bar  for  question  10,  as 
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shown  in  Chart  1,  indicates  the  median  value  was  5  The  respondents  tended  to  "agree"  that 
LeatherNet  improved  their  understanding  of  most  of  the  other  aspects  of  conducting  an 
estimate  of  the  situation,  particularly  friendly  avenues  of  approach  (question  11)  ,  friendly 
fields  of  fire  (question  15),  and  shift/cease  fire  points  (question  19).  For  these  three  aspects, 
over  80%  of  all  test  participants  agreed  or  strongly  agreed  that  LeatherNet  improved  their 
understanding  of  them.  The  test  participants  had  a  generally  neutral  opinion  regarding  an 
improved  understanding  of  obstacles  (question  12),  enemy  avenues  of  approach  for 
counterattacks  (question  13),  and  ammunition  requirements  (qustion  23)  as  indicted  by  the 
median  values  of  three  to  each  of  these  questions. 

3.  Assessment  of  LeatherNet  Terrain 

When  asked  if  the  LeatherNet  terrain  closely  matched  the  Range  400  terrain 
(question  24),  81%  of  all  the  respondents  either  agreed  or  strongly  agreed  that  it  closely 
matched  the  real  terrain.  The  median  value  was  4  or  “agree.”  OiJy  one  respondent  disagreed. 
However,  it  should  be  noted  that  several  respondents  thought  that  the  best  way  to  improve 
LeatherNet  would  be  to  refine  the  terrain  resolution  to  better  show  the  microterrain.  Many 
test  participants  thought  that  the  terrain  tended  to  look  "cartoonish"  up  close  and  that  it  was 
not  possible  to  determine  the  type  of  ground  material  such  as  soil,  sand,  or  rock. 

4.  Assessment  of  LeatherNet  Information  Usage 

Charts  2-5  show  the  distribution  of  the  responses  to  survey  questions  25,  26,  41,  and 
42.  An  assessment  of  whether  critical  information  was  provided  by  LeatherNet  will  be  made 
followed  by  an  assessment  of  whether  that  information  influenced  the  final  attck  plan.  The 
questions  were  in  inverse  order  on  the  questionnaire  which  is  why  question  26  is  addressed 
here  first.  The  distribution  of  responses  to  question  26  regarding  whether  LeatherNet 
provides  critical  information  is  shown  in  Chart  2.  The  fact  that  65%  of  all  responses  were 
greater  than  or  equal  to  the  median  value  of  four,  indicates  there  is  general  agreement  that 
LeatherNet  provided  the  respondents  with  critical  information  that  was  previously  unknown 
to  them.  The  claim  that  they  were  able  to  gain  critical  information  is  important,  particularly 
in  light  of  the  fact  that  the  respondents  had  ample  alternative  sources  of  information 
regarding  the  exercise.  Despite  the  use  of  map  studies,  aerial  photo  studies,  and  in  many 
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cases,  gathering  information  from  fellow  commanders  who  had  previously  run  the  range  and 
even  physically  walking  the  ground  prior  to  the  exercise,  they  were  still  able  to  obtain 
previously  unknown  information  from  LeatherNet.  The  reason  for  the  knowledge  gain  is 
probably  due  to  the  systems  ability  to  present  visually  that  information  normally  obtained 
by  one's  imagination,  such  as  weapons  &e  fans. 


The  responses  to  question  25  shown  in  Chart  3,  however,  indicate  that  there  was  only 
moderate  tendency  for  the  additional  information  provided  by  LeatherNet  to  influence  the 
final  plan  of  attack.The  median  value  was  four,  with  52%  of  respondents  claiming 
LeatherNet  influenced  their  plans.  The  fact  that  more  respondents  claimed  to  have  gained 
critical  information  than  changed  their  plans  based  on  that  information  may  be  explained  in 
part  by  the  lack  of  freedom  to  alter  plans  due  to  range  regulations  and  other  constraints 
placed  upon  the  participants.  An  alternate  explanation  would  be  some  lack  of  user 
confidence  in  the  system,  but  responses  to  questions  41, 42,  and  44  fail  to  support  this  idea. 
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Q25 

Chart  3.  Assessment  of  LeatherNet’s  Influence  on  the  Final  Plan  of  Attack 

The  distribution  of  responses  for  question  41,  shown  in  Chart  4,  indicates  some 
respondent  discomfort  in  relying  upon  LeatherNet  to  evaluate  their  plan  of  attack  because 
they  were  unfamiliar  with  the  system.  While  the  median  response  of  three  indicates  neither 
comfort  nor  discomfort  as  the  central  tendency,  1 9%  of  the  respondents  indicated  they  did 
have  some  degree  of  discomfort.  However,  the  responses  to  question  42,  shown  in  Chart  5, 
indicate  that  nearly  all  respondents  disagree  or  express  neither  agreement  nor  disagreement 
with  the  statement  that  they  "did  not  change  their  plan  of  attack  because  they  felt  they  could 
not  trust  LeatherNet  results  for  use  in  a  real  exercise."  Only  one  respondent  agreed  that  he 
did  not  alter  his  plan  of  attack  because  he  felt  he  could  not  trust  LeatherNet,  and  none 
strongly  agreed.  The  assumption  here  could  be  that,  while  the  system  was  unfamiliar  to  the 
users,  the  presentation  of  information  appears  to  be  authoritative  and  reliable  enough  for 
planning  purposes.  Before  reaching  this  conclusion,  however,  it  would  be  wise  to  consider 
that  Marine  ofiBcers  may  not  desire  to  reveal  that  they  did  not  change  a  plan  of  attack  in  the 
face  of  information  indicating  that  that  was  the  proper  course  of  action. 
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Q41 

Chart  4.  Responses  to  Question  41  Regarding  Whether  the  Respondents  Felt 
Uncomfortable  Using  LeatherNet  to  Evalauate  a  Plan  of  Attack 


8 


Q42 

Chart  5.  Responses  to  Question  42  Concerning  Whether  the  Respondents  Failed 
to  Use  LeatherNet  Information  in  the  Final  Plan  of  Attack  Due  to  Lack  of  Trust 
in  the  System 
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Spearman’s  Rho  was  used  to  determine  if  a  correlation  exists  between  the  user’s 
experience  level  and  the  degree  to  which  LeatherNet  influenced  the  user’s  flnal  attack  plan 
and  provided  previously  unknown  critical  information.  Spearman’s  Rho  is  used  to  test  the 
null  hypothesis,  that  there  is  no  relationship  between  a  participant’s  experience  and  the 
degree  to  which  LeatherNet  provides  critical  information  and  influences  the  user’s  plan  of 
attack.  It  was  expected  that  the  alternative  hypothesis,  that  there  is  a  relationship  between 
a  user’s  experience  and  the  degree  to  which  LeatherNet  influences  the  plan  of  attack  and 
provides  critical  information,  would  prove  to  be  true.  This  expectation  was  based  on  the 
typical  tendency  for  more  experienced  personnel  to  rely  upon  their  own  skills  and  methods 
rather  than  adopt  a  planning  tool  of  unknown  value.  Upon  conducting  the  test,  Spearman's 
Rho  values  of  .0287  and  -.1302  were  calculated  for  questions  25  and  26  respectively.  A 
Spearman's  Rho  of  0  signifies  no  correlation  between  variables,  whereas  a  Spearman's  Rho 
of  plus  or  minus  1  signifies  total  correlation.  It  can  easily  be  seen  that  no  significant 
correlation  exists  between  a  user's  experience  and  LeatherNet's  ability  to  influence  the  his 
final  attack  plan  and  provide  critical  information  to  him.  Therefore,  we  fail  to  reject  the  null 
hypothesis.  In  fact,  to  be  95  %  confident  that  the  null  hypothesis  is  rejected  only  when  it  is 
indeed  false,  a  P-value  of  .05  or  smaller  must  be  attained.  The  P-values  of  the  calculated 
Spearman's  Rho  were  .878  and  .485  respectively. 

The  implication  of  these  findings  is  that  LeatherNet,  as  used  in  these  tests,  has  value 
as  a  mission  planning  tool  to  the  test  participants  regardless  of  their  experience  level  and 
familiarity  with  Range  400.  For  training  purposes,  the  indications  are  that  company  level 
stafifs,  of  a  wide  range  of  experience  levels,  can  improve  their  understanding  of  an  exercise 
through  simulation,  and  that  LeatherNet  training  adds  value  to  the  CAX  training  program. 

5.  Assessment  of  LeatherNet  as  a  Mission  Planning  and  Rehearsal  Tool 

Charts  6-9  on  the  following  pages  display  the  distribution  of  responses  to  questions 
27-30  of  the  user  survey. 
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Chart  6.  Assessment  of  LeatherNet’s  Potential  as  an  Effective  Mission  Planning 
Tool 
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Chart  7.  Assessment  of  LeatherNet’s  Potential  to  be  an  Effective  Mission  Rehearsal 
Tool 
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Chart  8.  Assessment  of  Whether  LeatherNet  Provides  Vital  Information  in 
Evaluating  Tactical  Alternatives 
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Chart  9.  Assessment  of  LeatherNet’ s  Potential  to  be  an  Effective  Tool  for  Training 
Company  Commanders  and  Their  Staffs 
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Overall,  the  test  participants  believe  that  LeatherNet  demonstrates  the  potential  to 
be  an  effective  mission  planning  and  rehearsal  tool,  that  it  is  effective  for  training  company 
commanders  and  their  staffs,  and  that  it  provides  vital  information  for  evaluating  tactical 
alternatives.  Less  than  10%  of  all  respondents  expressed  a  negative  or  neutral  response  to 
questions  27,  29,  and  30.  The  highest  number  of  neutral  and  negative  responses,  totaling 
25%  of  all  responses,  were  given  to  question  28  regarding  LeatherNet's  potential  to  be  an 
effective  mission  planning  tool.  This  is  probably  due  to  the  fact  the  participants  did  not 
rehearse  their  plans  with  the  system,  and  were  therefore  only  speculating  about  the  systems 
potential  in  this  regard  based  on  the  credibility  of  the  other  functions  they  observed. 

6.  Assessment  of  Possible  Alternate  Uses  for  LeatherNet 

The  respondents  answers  to  questions  3 1  -34  indicate  wide  agreement  that  use  of 
LeatherNet  at  their  home  bases,  with  more  time  to  use  the  system,  and  with  more  of  their 
personnel  exposed  to  the  system,  would  be  beneficial.  Furthermore,  they  agree  that 
LeatherNet  would  be  of  more  value  if  other  sources  of  information  concerning  Range  400 
were  not  available,  such  as  having  the  abihty  to  physically  walk  the  range  and  get  advice 
from  other  personnel  who  had  completed  a  Range  400  exercise.  Only  3-6%  of  all 
respondents  disagreed  with  any  of  these  four  statements. 

7.  Assessment  of  the  Most  and  Least  Useful  Aspects  of  LeatherNet 

The  most  useful  features  of  LeatherNet,  according  to  the  respondents  answers  to 
question  36  are  listed  below.  The  features  are  listed  in  decreasing  order  of  importance  as 
determined  by  the  number  of  respondents  Usting  each  feature.  The  number  of  respondents 
listing  each  feature  is  given  in  parenthesis.  It  is  important  to  remember  that  the  users  were 
not  exposed  to  the  full  fimctionality  of  the  system. 

•  The  ability  to  display  both  fiiendly  and  enemy  fields  of  fire  and  engagement  areas 

(15) 

•  The  ability  to  "walk  the  terrain"  for  reconnaissance  purposes  (12) 

•  The  ability  to  visually  display  intelligence  on  the  enemy  (4) 

•  The  ability  to  display  avenues  of  approach  (3) 
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The  least  useful  features  according  to  the  respondent's  answers  to  question  37  are  as 
follows: 


•  Too  many  overlays  and  multi-color  templates  that  are  confusing  (5) 

•  The  possibility  of  depending  on  simulators  for  training  at  the  expense  of  training 
in  a  real  environment  (3) 

•  The  presentation  of  too  much  information,  too  fast,  to  be  useful  (2) 

•  Lack  of  details  on  microterrain  such  as  resolution  and  surface  type  (2) 

Below  are  the  user's  responses  to  question  35,  listing  in  decreasing  order  of 
importance,  the  users  opinions  on  how  to  improve  the  system.  These  opinion  are  based  on 
having  only  used  LeatherNet  for  mission  planning  and  briefing. 

•  Develop  an  ability  to  war  game  against  a  realistic  enemy  (  7) 

•  Refine  the  terrain  to  give  more  details  on  the  microterrain  and  represent  different 
t3q)es  of  ground  material  (5) 

•  Provide  more  details  on  the  enemy  forces  (4) 

•  Expand  the  terrain  database  to  cover  other  areas  (2) 

•  Use  at  other  bases  (2) 

It  should  be  noted  that  with  the  exception  of  the  microterrain  issue,  LeatherNet 
already  provides,  or  will  soon  provide,  all  of  the  above  listed  functions. 

8.  Determine  the  Most  Important  LeatherNet  Capabilities  to  Overcome  the 
Perceived  Limitations  of  Live  Fire  Training 

The  respondents  ranking,  in  decreasing  order  of  importance,  of  LeatherNet's 
perceived  ability  to  overcome  the  perceived  limitations  of  live  fire  training  are  displayed  in 
list  below. 


•  Ability  to  war  game  against  a  realistic,  responsive  enemy 
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•  Ability  to  obtain  quantitative  feedback  on  the  effectiveness  of  weapons 
employment 

•  Ability  to  train  on  simulated  terrain  in  the  abscence  of  real  terrain 

•  Ability  to  reconstruct/replay  an  exercise  for  detailed  debrief 

•  Ability  to  conduct  the  same  exercise  under  varying  conditions  or  to  modify  a  plan 
and  evaluate  the  outcome 

•  Ability  to  conduct  "danger  close'  fire  support 

•  Ability  to  evaluate  a  subordinate's  performance 

•  Simulated  casualty  assessment 

The  order  was  determined  by  the  median  value  of  all  respondent  rankings.  For  those 
characteristics  with  the  same  median  value,  the  mean  value  of  the  respondent's  rankings  was 
used  to  determine  the  order.  The  ability  to  war  game  against  a  realistic,  responsive  enemy 
was  perceived  to  be,  by  a  wide  margin,  the  most  important  characteristic  provided  by 
LeatherNet  that  overcome  the  limitations  of  live  fire  training. 

9.  Assessment  of  the  Respondent's  Attitude  Towards  LeatherNet 

Question  43  asks  the  test  participants  if  they  generally  had  a  positive  attitude  toward 
LeatherNet  prior  to  using  the  system  and  question  44  asks  them  if  their  attitude  is  more 
positive  after  using  the  system.  As  might  be  expected,  the  user's  attitude  is  generally  neutral 
towards  LeatherNet  prior  to  using  the  system  which  indicates  a  lack  of  bias  for  or  against  the 
system  prior  to  use.  In  fact,  none  of  the  respondents  disagreed  with  the  statement  that  their 
attitude  was  generally  positive  although  nearly  50%  expressed  a  neutral  opinion.  The 
respondent's  attitude  towards  LeatherNet  was  much  more  positive  overall  after  exposure  to 
the  system  with  77%  of  the  test  participants  indicating  a  more  positive  attitude.  Only  one 
participant  stated  that  his  attitude  was  less  positive  after  exposure.  The  attitudes  expressed 
here  tend  to  indicate  high  user  acceptance  of  LeatherNet  as  a  mission  planning  and  briefing 
tool.  The  distribution  of  the  responses  to  questions  43  and  44  are  shown  in  Charts  10  and  1 1 
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respectively. 


Q43 

Chart  10.  Assessment  of  the  Respondents  Attitude  Towards  LeatherNet  Prior  to 
Using  the  System 


20 


Q44 

Chart  11.  Assessment  of  Respondent’s  Attitude  Towards  LeatherNet  After 
Exposure  to  the  System 
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B.  USER  PERFORMANCE  EVALUATION 


1.  Performance  Testing  Results 

A  total  of  12  rifle  companies  firom  three  regular  infantry  battalions  and  one  reserve 
infantiy  battalion  had  their  performance  evaluated  on  Range  400  in  connection  with  this 
evaluation  of  LeatherNet.  One  rifle  company  from  each  battalion  served  as  the  control  group 
and  received  no  LeatherNet  training.  Therefore,  four  control  units  and  eight  experimental 
units  were  sampled.  Using  the  t  test  to  test  the  null  hypothesis,  that  no  difference  exists 
between  the  performance  means  of  the  two  groups,  revealed  no  statistically  significant 
difference.  The  t  value  was  calculated  to  be  .  179  with  n  - 1  degrees  in  each  sample  so  that 
the  total  degrees  of  fi’eedom  equaled  10.  Using  a  t  distribution  table,  this  value  was  compared 
to  the  critical  value  that  ensures  a  95%  confidence  that  we  reject  the  null  hypothesis  only 
when  it  is  indeed  false.  That  critical  value  was  2.228.  The  t  value  must  be  greater  than  the 
critical  value  to  reject  the  null  hypothesis.  Therefore,  we  must  fail  to  reject  the  null 
hypothesis  at  this  level  of  confidence. 

The  control  units  averaged  78.5%  of  their  objectives  successfully  completed  while 
the  experimental  units  achieved  79.6%.  The  small  sample  size  can  skew  the  results 
particularly  in  the  case  of  the  control  units.  Three  of  the  control  units  averaged  71%  of  their 
objective  achieved  but  the  fourth  control  unit  achieved  100%.  The  ability  of  this  one  unit  to 
influence  the  overall  results  suggests  the  need  for  a  larger  sample  size  to  ensure  that  the 
sample  closely  reflects  the  true  characteristics  of  the  population  as  a  whole.  If  this  one  unit 
was  eliminated  fi-om  the  study,  the  t  test  would  reveal  a  statistically  significant  difference 
in  the  means  at  the  90%  confidence  level,  but  still  not  at  the  95%  level.  Due  to  the  fact  that 
the  performance  evaluation  relied  upon  the  opinion  of  a  single  SME,  the  possibility  also 
exists  for  the  bias  of  the  evaluator  to  have  a  profound  effect  on  the  test  results  as  can  be  seen 
with  the  impact  of  one  unit  scoring  100%.  While  it  is  not  being  suggested  that  that  is  the 
case  here,  the  possibility  always  exists  in  a  subjective  evaluation. 

Other  reasons  may  exist  for  the  lack  of  variance  between  the  experimental  and 
control  units  performance.  As  mentioned  earlier,  lack  of  experimental  control  permitted 
external  influences  upon  both  the  experimental  and  control  units.  Access  to  additional 
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information  concerning  the  Range  400  exercise  by  the  control  units,  such  as  the  ability  to 
conduct  a  physical  reconnaissance,  would  tend  to  reduce  and  even  negate  any  information 
advantage  that  the  experimental  units  received  from  LeatherNet.  Additionally,  having  more 
and  even  better  information  doesn't  mean  that  the  information  is  put  to  its  best  use.  While 
the  user  surveys  indicate  that  a  large  portion  of  the  test  participants  gained  critical 
information,  and  that  the  information  influenced  their  final  attack  plans,  the  participants  still 
must  possess  the  necessary  skills  to  put  that  information  to  effective  use.  For  instance,  if  a 
test  participant  learned  from  LeatherNet  that  certain  avenues  of  approach  were  vulnerable 
to  enemy  fires,  the  participant  still  must  find  a  more  suitable  alternative  and  execute  that 
alternative  with  skill  to  realize  a  positive  effect.  Major  Newell,  reflecting  on  the  ability  of 
commanders  to  use  information  presented  to  them  during  CAX  training,  stated;  "giving  a 
man  tools  doesn't  make  him  a  carpenter,  but  rather  the  skill  with  which  he  uses  those  tools." 
Having  not  been  able  to  use  LeatherNet  to  war  game  their  plans,  the  test  participants  were 
left  with  only  their  normal  conventional  skills  and  imagination  to  finish  planning  and  execute 
their  plan  of  attack.  Finally,  the  test  participant  were  not  free  to  exercise  all  the  possible 
options  in  conducting  the  Range  400  exercise  due  to  safety  constraints  and  range  regulations 
which  limit  their  ability  to  take  advantage  of  the  information  gained  through  LeatherNet. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  MISSION  PLANNING  AND  BRIEFING  CAPABILITIES 

1.  User  Acceptance 

The  user  surveys  indicate  that  there  is  a  high  level  of  acceptance  among  the  test 
participants  for  LeatherNet  as  a  mission  planning  and  briefing  tool.  The  survey  respondents 
generally  agree  that  LeatherNet  improves  their  ability  to  conduct  an  estimate  of  the  situation 
and  provides  vital  information.  High  user  acceptance  is  supported  by  the  large  percentage 
of  users  who  said  the  system  provides  critical  information  and  that  the  information  they 
received  from  LeatherNet  influenced  their  final  attack  plan.  The  fact  that  50%  would  risk 
altering  their  plan  of  attack  based  on  information  fi'om  a  system  they  had  no  previous 
exposure  to  strongly  suggests  that  the  functionality  of  LeatherNet  for  mission  planning 
purpose  appears  both  authoritative  and  reliable.  Further  evidence  of  high  user  acceptance  is 
derived  from  the  fact  that  over  90%  of  all  respondents  agree  that  LeatherNet  has  the  potential 
to  be  an  effective  mission  planning  tool,  provides  vital  information  in  evaluating  tactical 
alternatives,  and  demonstrates  the  potential  to  be  an  eflfective  training  tool  for  training 
company  commanders  and  their  staffs.  Additionally,  over  75%  of  the  respondents  had  a 
more  positive  attitude  toward  the  system  after  seeing  it  in  operation. 

2.  Benefits  Derived  from  LeatherNet 

As  mentioned,  the  users  believe  that  the  system  provides  critical  information  that  was 
not  known  to  them  prior  to  using  LeatherNet.  The  significance  of  this  is  that  the  respondents 
had  ample  opportunity  to  gain  information  concerning  the  Range  400  exercise  from  many 
other  traditional  sources,  but  still  found  that  LeatherNet  provided  additional  unknown 
information  that  was  of  a  critical  nature.  The  respondents  indicated  the  ability  to  conduct 
"terrain  walks"  and  the  ability  to  display  both  enemy  and  friendly  fire  fans  to  be  the  most 
useful  aspects  of  using  LeatherNet  to  conduct  an  estimate  of  the  situation.  While  they  found 
LeatherNet  useful  for  gaining  information  regarding  most  elements  of  an  estimate  of  the 
situation  they  were  less  positive  or  expressed  neutral  sentiment  that  LeatherNet  helped  them 
understand  obstacles,  enemy  avenues  of  approach  for  counterattacks,  and  ammunition 
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requirements.  Finally,  the  test  participants  expressed  nearly  universal  belief  that  LeatherNet 
has  the  potential  to  be  an  effective  mission  planning  and  COA  analysis  tool,  and  that  it  is 
effective  for  training  unit  leaders  at  the  company  level. 

Unfortunately,  no  performance  benefits  could  be  quantified  from  using  LeatherNet 
according  to  the  results  from  the  Range  400  evaluations  despite  the  positive  user  appraisal 
of  the  system.  It  was  not  intended  that  empirical  testing  be  conducted  with  a  system  under 
development  and  without  the  ability  to  exercise  the  necessary  experimental  control.  Due  to 
the  cost,  resources,  and  time  necessary  to  perform  this  type  of  testing,  it  is  neither  desired 
nor  practical  at  this  point.  It  was  thought,  however,  that  early  user  testing  prior  to  full 
completion  would  provide  the  opportunity  for  user  feedback  while  the  system  remains  under 
development.  When  this  research  began  it  was  expected  that  a  more  comprehensive  test 
could  have  been  performed.  However,  developmental  issues,  resistance  to  using  CAX  units 
for  more  comprehensive  testing,  and  the  limited  availability  of  CAX  units  to  conduct  the 
tests  all  combined  to  limit  the  ability  to  fully  test  the  system.  Several  lessons,  however,  have 
been  gained  that  can  improve  the  quality  of  the  test  process  for  LeatherNet  in  the  future. 

B.  RECOMMENDATIONS 

1.  System  Testing 

The  Marine  Corps  is  continuing  to  increase  its  interest  and  investment  in  the  use  of 
simulation  technology  to  better  prepare  and  train  its  forces.  LeatherNet  represents  only  a 
small  portion  of  the  overall  effort  and  is  primarily  designed  to  be  a  demonstration  project 
with  a  secondary  purpose  of  providing  an  intuitively  appealing  training  tool  for  CAX  units. 
If  the  initial  feedback  from  the  survey  conducted  in  conjunction  with  this  research  is  any 
indication,  LeatherNet  has  a  good  potential  to  successfully  fulfill  a  role  as  a  valuable  training 
tool.  How  well  LeatherNet  may  demonstrate  its  capabilities  and  technologies  remmns  to  be 
seen.  While  LeatherNet  observers  may  have  a  qualitative  feel  that  the  system  can  provide 
effective  mission  planning,  mission  rehearsal,  and  "what  if  COA  analysis,  there  seems  to 
be  little  or  no  data  to  quantify  LeatherNet's  effectiveness.  Admittedly,  it  has  proven  to  be  a 
difficult  task  to  quantify  the  training  effectiveness  of  a  virtual  simulation  system,  particularly 
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one  designed  for  collective  training.  In  fact  it  is  hard  to  quantify  the  training  effectiveness 
of  collective  training  alone,  never  mind  evaluating  it  in  combination  with  simulation. 

In  order  to  quantify  the  training  effectiveness  of  virtual  simulators  such  as 
LeatherNet  and  TIES,  it  is  recommended  that  the  Marine  Corps  follow  the  Army’s  lead  in 
conducting  user  testing..  The  Army  is  aware  of  the  deficiency  in  quantifying  the  trdning 
effectiveness  of  its  training  simulators  and  has  committed  itself  to  conducting  the  most 
comprehensive  test  of  a  simulation  system  to  date  with  its  CCTT  system.  Although 
LeatherNet  may  be  primarily  a  demonstration  project,  the  best  way  to  demonstrate  the 
project  is  to  quantify  the  results.  The  ability  to  say  the  system  can  save  units  X  dollars  while 
improving  their  ability  to  successfully  complete  Y  training  objectives  by  Z  percent  is  a  much 
more  powerful  promotion  tool  than  simply  saying  we  believe  it  improves  training,  based  on 
user  feedback.  While  it  is  recognized  that  the  resources  for  testing  most  likely  will  only 
allow  for  a  much  scaled  down  version  of  the  Army’s  test  plans,  tests  can  still  be  designed 
to  provide  quantitative  data.  Such  data  can  be  used  to  provide  solid  justification  regarding 
future  M&S  investments. 

For  follow  on  tests,  it  is  recommended  that  dedicated  test  units  be  assigned  to  the 
project  during  the  test  period,  rather  than  using  units  distracted  by  other  training  and 
command  commitments.  Undedicated  test  units  will  tend  to  give  the  test  process  lower 
priority  than  their  operational  demands,  and  the  ability  to  conduct  the  test  properly  will 
suffer  accordingly.  Secondly,  experimental  control  needs  to  be  exercised  over  the  test  units 
to  eliminate  outside  sources  of  influence  that  can  corrupt  the  results.  Neither  the  control  nor 
the  experimental  units  should  have  access  to  outside  information  and  assistance  other  than 
that  information  and  assistance  which  the  evaluators  desire  to  allow  each  unit.  Particularly, 
in  the  case  of  an  exercise  such  as  Range  400,  neither  group  should  be  allowed  to  do  a 
reconnaissance  of  the  enemy  held  terrain  nor  collect  advice  fi-om  others  who  have  just 
completed  the  exercise.  Denying  the  control  units  the  ability  to  collect  intelligence  not 
normally  available  on  a  battlefield  will  more  likely  show  how  much  more  valuable  a  virtual 
simulator  can  be  in  preparing  for  an  exercise  compared  to  preparing  without  if  it  is  in  fact 
more  valuable.  Conducting  a  true  experiment,  consisting  of  a  pre-test  to  determine  skill 
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levels,  training  on  LeatherNet,  and  a  post-test  to  determine  post-training  skill  levels  for  the 
experimental  group  and  a  pre-test  and  post-test  for  the  control  group  should  be  considered. 
Finally,  incorporation  of  instrumented  ranges,  such  as  those  being  developed  for  the  Emerald 
Light  Project  to  assess  the  performance  of  the  units  in  both  the  pre-test  and  post-test  phases 
should  also  be  considered.  Instrumented  ranges  can  provide  data  analysis  of  position  and 
tracking  of  indirect  fires  and  position  and  trigger-pulls  of  ground  entities  and  direct  fires. 
Developing  a  methodology  to  use  this  analysis  to  rate  performance  provides  the  possibility 
to  eliminate,  to  a  large  degree,  the  subjective  nature  of  evaluating  collective  training. 

While  this  type  of  an  experiment  will  most  likely  be  expensive  and  time  consuming 
to  conduct,  the  units  will  be  able  to  gain  worthwhile  training  while  providing  valuable  test 
data.  Such  data  can  also  assist  the  Marine  Corps  modeling  and  simulation  effort  by  providing 
information  on  how  to  improve  our  training  simulators  and  where  to  focus  our  M&S 
development  and  procurement  dollars  . 

2.  Expansion  of  M&S  Resources 

The  use  of  virtual  simulation  systems  holds  great  promise  to  provide  unique 
opportunities  to  overcome  the  current  and  future  obstacles  to  training  such  as  high  cost  and 
loss  of  training  areas.  The  great  m^ority  of  the  test  participants  agreed  that  it  would  have 
been  beneficial  to  have  been  able  to  use  the  LeatherNet  system  at  their  home  base,  to  have 
been  allowed  more  extensive  use  of  the  system,  and  to  include  more  of  their  personnel  in  the 
simulation  training.  To  make  these  desires  a  reality,  the  Marine  Corps  should  continue  to  the 
development  and  implementation  of  both  demonstration  and  training  sites  of  DIS  compliant 
simulators.  Doing  so  will  enable  Fleet  Marine  Force  (FMF)  commands  to  train  with  and  be 
exposed  to  what  promises  to  be  an  increasingly  important  training  alternative  for  combat 
forces.  Such  exposure  can  promote  acceptance  among  the  ranks  of  both  our  junior  and  senior 
leadership  and  perhaps  smooth  the  way  for  further  implementation. 

Expansion  of  the  terrain  databases  to  incorporate  other  areas,  particularly  littoral 
areas  is  essential  to  enhancing  the  training  value  of  virtual  simulators  such  as  LeatherNet  for 
the  Marine  Corps.  Current  efforts  to  develop  the  Littoral  Warfare  Training  Complex, 
including  instrumented  ranges,  data  processing  and  display  capabilities,  and  the  required 
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communication  infrastructure  is  a  step  in  the  right  direction.  Without  the  modeling  of  the 
littoral  terrain  as  well  as  the  associated  Marine  Corps  equipment,  tactics,  and  command  and 
control  characteristics,  the  Marine  Corps  risks  being  left  out  as  an  effective  participant  in 
joint  simulation  initiatives  and  exercises.  The  only  way  to  remain  relevant  in  these  initiatives 
is  to  devote  the  necessary  resources  to  develop  those  systems  and  functionality  that  cannot 
be  leveraged  from  the  other  services. 

The  Marine  Corps  faces  a  variety  of  training  constraints  that  could  potentially 
degrade  the  readiness  of  its  forces.  Investing  in  DIS  compliant  simulation  systems  will 
ensure  that  the  FMF  commands  have  access  to  world  class  training  opportunities  while 
preserving  resources,  in  the  long  run,  to  conduct  mission  essential  live  fire  training  exercises 
such  as  those  associated  with  the  CAX  program.  The  longer  the  Marine  Corps  takes  to 
investigate  the  various  simulation  options  and  invest  where  the  opportunities  appear  to  be 
the  most  promising,  the  longer  those  systems  will  be  denied  to  those  forces  that  might 
benefit  from  them. 
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APPENDIX  A.  LEATHERNET  USER  SURVEY 
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7  April  96 


Fellow  Marine, 

Your  unit  has  been  randomly  selected  to  participate  in  user  testing  of  LeatherNet.  An 
important  part  of  the  process  is  a  survey  of  your  opinion  toward  this  simulator  and  its  value 
as  a  mission  planning  and  mission  training  device.  The  survey  is  part  of  a  study  being 
conducted  at  the  Naval  Postgraduate  School  to  determine  the  value  of  virtual  reality 
simulators  in  helping  ground  units  to  train  and  prepare  for  combat  missions. 

Currently,  LeatherNet  represents  the  only  operational  simulator  of  its  kind  in  the  U.S. 
Marine  Corps.  Therefore,  the  results  of  this  survey  are  important  because  they  are  the  only 
data  available  concerning  what  the  "Grunts"  think  of  the  system. 

All  responses  will  be  kept  confidential.  Only  the  survey  data  will  be  compiled  and  no 
personal  or  unit  data  will  be  released  from  this  survey. 

Please  answer  all  questions  to  the  best  of  your  ability.  Thank  you  for  your  cooperation. 


Tracy  R.  Hague 
Major  USMC 
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RANGE  400  /  LEATHERNET  LAB  QUESTIONNAIRE 


COMPANY: _ 

BILLET: _ 

RANK: _ 

CAX  #: _ 

DATE  IN  LEATHERNET  LAB:, 
DATE  ON  RANGE  400: _ 


L  Experience  Questions 


1 .  How  many  years  of  active  military  service  do  you  have? 

O  Fewer  than  3  years 
O  3  to  6  years 
O  6  to  9  years 
O  9  to  12  years 
O  More  than  12  years 

2.  How  long  have  you  held  your  current  billet? 

O  Fewer  than  6  months 
O  6  to  12  months 
O  12  to  18  months 
O  18  to  24  months 
O  Longer  than  24  months 

3.  Have  you  personally  run  range  400  anytime  prior  to  this  CAX? 

O  Yes 
O  No 

If  yes,  when  did  you  last  run  range  400? _ 

4.  How  often  does  your  company  conduct  live  fire  mid  maneuver  exercises? 
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O  Fewer  than  once  a  year 
O  1  to  3  times  a  year 
O  4  to  6  times  a  year 
O  7  to  10  times  a  year 
O  More  than  10  times  a  year 
O  Don't  know 

5.  How  would  you  rate  your  company's  overall  tactical  proficiency? 

O  Far  below  average 
O  Below  average 
O  Average 
O  Above  average 
O  Far  above  average 

6.  Did  you  physically  walk  any  of  the  Range  400  terrain  forward  of  Machine  Gun 
Hill  prior  to  the  live  fire  exercise? 

O  Yes 
O  No 


7.  Did  you  receive  advice  on  how  to  conduct  the  Range  400  exercise  fi'om  other 
Marines  in  your  unit  who  had  previous  experience  on  Range  400? 


O  Yes 
O  No 


8.  Did  you  observe  other  units  conduct  the  Range  400  exercise  during  the  current 
C AX  prior  to  your  company's  Range  400  exercise? 


O  Yes 
O  No 
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n.  LeatherNet  User  Questions 


A.  Ability  to  conduct  an  estimate  of  the  situation 

Questions  8  through  23  refer  to  the  statement  below. 


The  LeatherNet  lab  improved  my  understanding  of  the  following  information: 


Enemy  situation 

9.  Enemy  positions 

Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 

10.  Enemy  fields  of  fire 

Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 

Terrain 

1 1 .  Friendly  avenues 

of  approach 

Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 
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The  LeatherNet  lab  improved  my  understanding  of  the  following  information: 

12.  Obstacles 


Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 

13.  Enemy  avenues  of  approach  for  counterattacks 

Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 

14.  Potential  friendly  firing  positions 

Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 

Fire  support 

15.  Fields  of  fire  for  fiiendly  weapons  from  potential  firing  positions 

Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 

16.  Range  fans  for  fnendly  weapons  from  potential  firing  positions 

Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

Disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 
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The  LeatherNet  lab  improved  my  understanding  of  the  following  information: 

17.  Mortar  sheafs  ability  to  suppress  enemy  positions 


Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 

18.  Machine  gun  target  coverage 

Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 

19.  Shift/cease  fire  points 

Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 

20.  Weapon  safety  fans 

Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 

Time/Space 

21.  Time  exposed  to  enemy  fire  along  avenues  of  approach 

Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 
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The  LeatherNet  lab  improved  my  understanding  of  the  following  information: 

22.  Time  required  to  execute  movements  along  avenues  of  approach 


Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

Logistics 

23.  Ammunition  requirements 

Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

B.  Overall  effectiveness  of  LeatherNet 

24.  The  simulated  terrain  in  LeatherNet  closely  matched  the  Range  400  terrain. 


Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

25.  The  LeatherNet  lab  influenced  my  final  attack  plan. 

Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

26.  The  LeatherNet  lab  provided  critical  information  that 
unknown  to  me. 

was  previously 

Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 
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27.  LeatherNet  has  the  potential  to  be  an  effective  mission  planning  tool. 


Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

2f?  LeatherNet  has  the  potential  to  be  an  effective  mission  rehearsal  tool. 

Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

29.  LeatherNet  provides  vital  information  in  evaluating  tactical  alternatives. 

Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

30.  LeatherNet  demonstrates  the  potential  to  be  an 
commanders  and  their  staffs. 

effective  tool  for  training  company 

Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

31.  The  LeatherNet  lab  would  be  more  useful  if  we  could  use  it  at 
just  prior  to  CAX  . 

our  home  base 

Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

32.  The  LeatherNet  lab  would  be  more  useful  if  we  had  more  time  to  use  the 
system  to  evaluate  our  plans. 

Strongly 

disagree 

O 

Disagree 

O 

Neither  agree 
nor  disagree 
O 

Agree 

O 

Strongly 

agree 

O 
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33.  The  LeatherNet  lab  would  be  more  useful  if  more  personnel  in  my  unit  had  the 
opportunity  to  use  the  system. 


Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

The  LeatherNet  lab  would  have  been  of  more  value  in  preparing  for  Range  400  if 
we  did  not  have  other  means  of  obtaining  information  concerning  the  exercise  such 
as  our  own  reconnaissance  or  our  fellow  Marine's  advice. 

Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

35.  What  other  changes  can  be  made  to  make  LeatherNet  more  useful? 


36.  What  were  the  most  useful  features  of  LeatherNet? 


37.  What  were  the  least  useful  features? 
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38.  Please  rank  the  following  capabilities  in  order  of  importance  (  1  to  indicate  the 

most  important,  8  to  indicate  the  least  important)  Please  note  that  you  may 
have  not  been  exposed  to  all  of  these  capabilities.  If  that  is  the  case,  your  ranking 
should  reflect  how  important  you  believe  it  would  be  to  have  these  capabihties. 

A.  Ability  to  war  game  against  a  realistic,  responsive  enemy  _ 

B.  Quantitative  feedback  on  effectiveness  of  weapons  employment  _ 

C.  Ability  to  reconstruct/replay  an  exercise  for  detailed  debrief  _ 

D.  Ability  to  conduct  "danger  close"  fire  support  training  _ 

E.  Ability  to  evaluate  a  subordinate's  performance  _ 

F.  Ability  to  train  on  simulated  terrain  in  the  absence  of  the  real 

terrain  (ie:  aboard  a  ship)  _ 

G.  Ability  to  conduct  the  same  exercise  under  varying  conditions 

or  to  modify  a  plan  and  evaluate  its  outcome  _ 

H.  Simulated  casualty  assessment  _ 

39.  It  would  have  been  beneficial  to  evaluate  our  plan  of  attack  using  all  of  the  features 
listed  in  question  37. 


Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

nor  disagree 

agree 

O 

O 

O 

O 

O 

40.  What  other  features  would  you  like  to  see  developed  for  LeatherNet? 
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41 .  I  felt  uncomfortable  relying  on  LeatherNet  to  evaluate  my  plan  of  attack 
because  I  was  unfamiliar  with  computer  simulators. 


Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

I  did  not  change  my  plan  of  attack  because  I  was  not  sure  I  could  trust 
results  of  the  LeatherNet  lab  in  a  real  exercise. 

the 

Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

My  attitude  toward  LeatherNet 
LeatherNet  lab. 

was  generally  positive  prior  to  conducting  the 

Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

My  attitude  toward  LeatherNet  is  more  positive  now  that  I  have  seen  it  in 
operation. 

Strongly 

disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly 

agree 

O 

O 

O 

O 

O 

Please  return  this  survey  to  Major  Newell  or  the  survey  coordinator. 
Thank  you  again  for  your  cooperatio 


74 


APPENDIX  B.  RANGE  400  EVALUATION  FORMS 
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UNIT 


Date 


Bases  of  Fire/firing  position  selection. 
Appropriate  ranges  to  enemy  positions. 


Heavy  machine  guns 

YES 

NO 

Machinegun  section 

YES 

Dragons. 

YES 

NO 

Rifle  platoon  bases  of  fire. 

YES 

NO 

COMMENTS: 


Route  selection. 

Covered/concealed  routes  used  to  occupy  Base  of  Fire/firing  positions? 


Heavy  machine  guns. 

YES 

NO 

Machinegun  section. 

YES 

NO 

Dragons. 

YES 

NO 

Rifle  platoon  bases  of  fire. 

YES 

NO 

COMMENTS: 
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Covered  and  concealed  avenue  of  approach  used  for  assault  element. 


YES  NO 


COMMENTS: 


SUPPORT  OF  MANEUVER. 

Suppressive  fires  used  in  support  of  occupation  of  Base  of  Fire/firing  positions. 


Machinegun  section. 

YES 

NO 

Dragons. 

YES 

NO 

Rifle  platoon  bases  of  fire. 

YES 

NO 

COMMENTS: 


Suppressive  fire  delivered  on  all  enemy  positions  that  could  engage  exposed 
movement  of  the  assault  element. 


Breach  of  minefield  and  movement  into  wash. 

YES 

NO 

Breach  of  wire. 

YES 

NO 

Assault  of  western  position. 

YES 

NO 

Assault  of  center  position. 

YES 

NO 

Assault  of  CP  position. 

YES 

NO 
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COMMENTS: 


Obscuration  delivered  on  enemy  positions  that  could  deliver  fire  on  obstacle  breaches. 


Breach  of  minefield. 

YES 

NO 

Breach  of  wire. 

YES 

NO 

COMMENTS: 


Counterfire  delivered  on  enemy  mortar  position. 

At  beginning  of  attack/pre-H-Hour.  YES  NO 

During  breach  of  wire  obstacle.  YES  NO 

COMMENTS: 


Delivery  of  suppressive  fires  effective. 
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Adequate  coverage  of  enemy  positions  (i.e.  mortar  sheafs). 
Appropriate  machinegun  gunnery  techniques  employed. 


YES 

NO 

YES 

NO 

COMMENTS: 


Fires  shifted/ceased  at  appropriate  times  (by  the  unit,  as  close  as  possible  without 
violating  safety  fans). 


YES  NO 


COMMENTS: 


No  unnecessary  fires  (fires  on  enemy  positions  unable  to  engage  friendlies). 

YES  NO 


COMMENTS: 


Firing  units  did  not  run  out  of  ammunition  before  their  mission  was  complete. 
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60mm  mortars 

YES 

NO 

81mm  mortars 

YES 

NO 

Machinegun  section 

YES 

NO 

Heavy  machine  guns 

YES 

NO 

Assault  platoons 

YES 

NO 

COMMENTS: 


Appropriate  positions  were  selected  to  observe/adjust  mortar  fires  (STA/FST). 

YES  NO 


COMMENTS: 
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